Environment for local development with remote server

[imso077]

Currently it is not straightforward to work/develop on the PWA theme when we have an already running Magento instance in a remote server.

Would be noce to have an enviroment where we run the PWA theme locally, and we connect it to the remote server (graphql etc...)

I think this will help when migrations are done, where the PWA have to built in specifc store, with specific set of products cms pages etc.

[alfredsgenkins]

Not currently near a PC, but what is needed:

1. Run the npm run watch and make sure the fallback path in webpack config is correct.
2. Configure the nginx, just like in deploy/frontend/nginx but replace all hosts and proxy_pass to your instance.

I will take a look at this, and probably share a docker-compose as soon as I will be available :)

[alfredsgenkins]

Here is my nginx config. What I do:

1. Run npm run watch - make sure there is a fallback (source) directory present. You can change the path to it from in the webpack config.
2. Make sure you have nginx and add the config file to it:

server {
  listen 80;
  server_name _;
  resolver 127.0.0.11;

  # Route all traffic to app
  location / {
    proxy_pass http://localhost:3003;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_set_header Host              $http_host;
    proxy_set_header X-Real-IP         $remote_addr;
    proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
    proxy_redirect off;
    proxy_read_timeout 600s;
  }

  location ~* /graphql {
    proxy_pass http://mysite.com:80;
    proxy_set_header Host             mysite.com;
    proxy_set_header X-Real-IP         $remote_addr;
    proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
    proxy_redirect off;
    proxy_read_timeout 600s;
  }

  location ~* /(static|media|admin) {
    proxy_pass http://mysite.com:80;
    proxy_set_header Host              mysite.com;
    proxy_set_header X-Real-IP         $remote_addr;
    proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
    proxy_redirect off;
    proxy_read_timeout 600s;
  }
}

replace mysite.com with your instance.

[alfredsgenkins]

Sorry no time for docker-compose to prepare, but please use this to proceed. If feeling brave - contribute the docker-compose for such setup, will be very-very happy to merge in!

[imso077]

@alfredsgenkins Thanks for the info.
I've set up now my environment, with this help.

FYI: I'm not using any of the docker instances, as I already have a dev server, where Varnsh etc all set up. And in my local I've the code base, therefore modified only the Webpack config for the correct path, and used the nginx routing what you given to me. (Nginx runs locally on my machine already)

I've found this setup easier in our case.

[alfredsgenkins]

And what needs to be changed in webpack.development.config.js:

const projectRoot = path.resolve(__dirname, '..', '..'); // stays the same
const magentoRoot = path.resolve(projectRoot, '..', '..', '..', '..', '..'); // not needed !
const fallbackRoot = path.resolve(magentoRoot, 'vendor', 'scandipwa', 'source'); // clone the same version of the theme as is installed on server, and point the "fallbackRoot" into it :

I do belive this setup is easier when developing the FE. For BE development it is required to have Magento running locally - this is were the development server might help :)

[astik]

@imso077 , i had the same objective as you : working with a remote magento and with minimal tools. Here is the result of my analysis : https://github.com/astik/scandipwa-init-front.

First step, you initialize a project (with init.sh), then you follow guidelines from README.md which are quite standard for node relative project (npm install, npm run ...).

It is working with ScandiPWA base-theme v2, i didn't try yet with v3.

@alfredsgenkins having those kind of setup is great as it is fast to setup and very light to run. To really go further with a great developer experience, i can see 2 things to enhance:

• to have base-theme as a npm dependency, so that the fallback plugins could look into the node_modules/@scandipwa/base-theme dependency for fallbacks), no need to have vendor folder next to our development
• to have all dev script as part of a packaged tool like what is done with create-react-app: a single dependency with embedded webpack conf (maybe expending current create-react-app like this: https://create-react-app.dev/docs/alternatives-to-ejecting)

[alfredsgenkins]

Oh, my god, how did I miss your comment? @astik amazing job! Let me look through and publish similar solution! Great job! 👏

[astik]

I tried some iterations over a custom react-script (from create-react-app). FWIW here are the results of the analysis:

Goals towards custom create-react-app

• use as much as possible create-react-app

To achieve this goal:

• create all minimum initial files as boilerplate: define a CRA template
• build as a Magento theme: custom webpack build configuration
• having a folder structure similar to create-react-app: refactor base-theme

For allow extending base-theme:

• being able to fallback on base-theme for js and css app file, for service worker file: custom webpack plugin
• manage file resolution through aliases: jsconfig.json? babel specific configuration? webpack custom resolver?

For i18n:

• manage i18n: custom webpack plugin
• add fallback for i18n: first look into MY_THEME/i18n then look into node_modules/base-theme/i18n (this can easily be done into mapTranslationsToConfig): configure custom i18n webpack plugin
• extract i18n key used in the project: custom webpack plugin

In index.html:

• we should have %PUBLIC_URL% instead of `<%= htmlWebpackPlugin.options.publicPath %>``
• JS and CSS coming from webpack should not be defined as they are injected automatically in the HTML file

Ideal file structure

i18n folder

Keys defined in this folder will override existing one in the base theme.
It extends base-theme i18n folder in order not to have to copy/paste existing file.

Public folder

It contains static file that should not be bundled.

src folder

It contains every application file: JS, CSS/SCSS, images, ... that goe through the bundling process

src-magento folder

It contains every files relative to Magento theme definition.
There is no application file in this folder, no frontend relative information.

Retro engineering for current dev process

• ScandiPWA has 2 entries points: src/app/index.js and src/sw/index.js
  -> this is needed
• Fallback plugin acts as an overlay on base-theme: look into custom theme and then fallback on base theme
  -> this is needed
• alias are available to resolve code inside custom theme and fallback on base theme (Style, Component, Route, ...)
  -> this is needed
• alias are available to resolve code inside base theme directly (SourceStyle, SourceComponent, SourceRoute, ...)
  -> this is needed
• SCSS are resolved without their extension
  -> maybe it would be better to be more explicit and we should add extension
• React dependency is injected automatically through Webpack's ProvidePlugin
  -> maybe it would be better to limit magic injection
• i18n is managed through a global variable __ which is injected automatically through Webpack's ProvidePlugin
  -> this is needed
• base theme needs specific babel plugin to work: @babel/plugin-proposal-class-properties, babel-plugin-module-resolver,
  -> is this really needed ? is it a prerequisite to define class component in order to allow extension ?

[astik]

Looking at v3-stable, there is some big changes that get us farther away from create-react-app:

• fallback plugin depends heavily on magento theme file structure
• scandipwa relies heavily on class component for inheritance and extension purpose
• middleware function wraps everything

Having a tool similar to CRA with scandipwa will need way more work than what i was expecting ='(

All the custom features should be well documented in order to easily maintain the build tools.

[astik]

I found a project that could be of help on using create-react-app: https://github.com/gsoft-inc/craco
It allows extending create-react-app by providing our own webpack configuration.
It would be great to have a scandipwa cli that encapsulate a craco cli that decorates create-react-app. That way, we would benefit of a create-react-app "standard" process and overrides would be more explicit as they would be separated from basic conf.

[alfredsgenkins]

Soo. V3 indeed is very dependant on composer. We are not the storefront in the end of the day. Extensions install directly from composer and require a single modification in scandipwa.json file. Thus, I consider the stand-alone installation possible, yet more complex than initially expected.

Following must be done to install V3 properly:

• Get / Clone for of ScandiPWA base. We will need src/composer.* from there. Use overrides from app/design/frontend/<VENDOR>/<THEME>
• Using composer, find a way to install dependencies without platform requirements (--ignore-platform-reqs)
• Install dependencies, i.e. npm ci (remind me if this changed @yeegor)
• Run watcher npm run watch, or other command by preference.

As seen we need composer and npm and git to run the ScandiPWA => I would suggest going for "light" docker setup. Is there a real need for it? Well, questionable, why not to go for "full" docker instead? Local machine performance... Well, true.

We are not a store-front. We are Magento theme. I am think if the composer plugins (not npm) was a right choice for us, but in the long run it means code installed from a single source. Both for BE and FE. What do you think?

[alfredsgenkins]

Theoretically, it can be packed into a single command. We are already working on similar command for docker-install, with single command it will run you the whole stack. The question is, do we need the same on local, without docker? We might make a questionnaire in Slack group.

[alfredsgenkins]

Most not experienced devs work on servers without deploy => they would want watcher-file-generator, which is actually really easy to make. Actually we should do it ASAP, even though this is bad approach people would love it.

[astik]

@alfredsgenkins, the idea is to remove magento specificities as much as possible from frontend developers. Having a specific folder structure following magento / composer convention is ok, no problem. Asking a frontend dev to install Magento, php, ... it's a lot i would prefer to avoid (to say the least).

if you look at https://github.com/astik/scandipwa-init-front, you'll find a very simple way to init a project. It is working (tested with scandipwapmrev.indvp.com as backend):

• simulate magento folder structure
• base theme mount as a submodule
• custom theme initialized thanks to base module

It is pretty straightforward and easy to setup.

So far, i'm running it correctly with base theme v2 but still got issue when replacing with v3 in my init script.

One thing with current code state, a lot of files relative to tooling (src/config and custom configuration .eslintrc, .stylelintrc). It might be interesting to find a way to externalize this in a separate npm dependencies (even though it is hidden behind a CLI)

[astik]

I just push new code on master to handle v3 (v2 has its own branch). I also add some specificities:

• adding a new host (scandipwa.local) is mandatory as it is hardcoded in webpack dev conf (it would be great to be able to set a custom isolated one, especially if we have multiple scandipwa projects in parallel)
• --env.BUILD_MODE=development needs to be set for starting dev watch (it was previously part of the script in v2)
• in order to make it work without composer in my starter, i removed dependencies inside scandipwa.json

(Disclaimer: i'm no php developer) For composer, i though i could run composer install from the theme project but i get an error:

➜  pwa git:(master) composer install
Loading composer repositories with package information
Updating dependencies (including require-dev)
Your requirements could not be resolved to an installable set of packages.

  Problem 1
    - The requested package magento/magento2-base could not be found in any version, there may be a typo in the package name.
  Problem 2
    - Installation request for scandipwa/cms-graphql ^1.3 -> satisfiable by scandipwa/cms-graphql[1.3.0].
    - scandipwa/cms-graphql 1.3.0 requires magento/module-cms-graph-ql * -> no matching package found.
  Problem 3
    - Installation request for scandipwa/klarna-graphql ^1.1 -> satisfiable by scandipwa/klarna-graphql[1.1.0].
    - scandipwa/klarna-graphql 1.1.0 requires magento/magento2-base ^2.3.2 -> no matching package found.
  Problem 4
    - scandipwa/persisted-query 2.3.0 requires magento/magento2-base ^2.3.5 -> no matching package found.
    - scandipwa/persisted-query 2.2.1 requires magento/magento2-base <2.3.5 -> no matching package found.
    - scandipwa/persisted-query 2.2.0 requires magento/magento2-base ~2.3.2 -> no matching package found.
    - scandipwa/persisted-query 2.1.2 requires magento/magento2-base ~2.3.2 -> no matching package found.
    - scandipwa/persisted-query 2.1.1 requires magento/magento2-base ~2.3.2 -> no matching package found.
    - scandipwa/persisted-query 2.1.0 requires magento/magento2-base ~2.3.2 -> no matching package found.
    - scandipwa/persisted-query 2.0.2 requires magento/magento2-base ~2.3.2 -> no matching package found.
    - scandipwa/persisted-query 2.0.1 requires magento/magento2-base ~2.3.2 -> no matching package found.
    - scandipwa/persisted-query 2.0.0 requires magento/magento2-base ~2.3.2 -> no matching package found.
    - Installation request for scandipwa/persisted-query ^2.0 -> satisfiable by scandipwa/persisted-query[2.0.0, 2.0.1, 2.0.2, 2.1.0, 2.1.1, 2.1.2, 2.2.0, 2.2.1, 2.3.0].

Potential causes:
 - A typo in the package name
 - The package is not available in a stable-enough version according to your minimum-stability setting
   see <https://getcomposer.org/doc/04-schema.md#minimum-stability> for more details.
 - It's a private package and you forgot to add a custom repository to find it

Read <https://getcomposer.org/doc/articles/troubleshooting.md> for further common problems.

As i want to work with frontend stuff only, i though i would remove all composer backend reference and only keep the frontend one defined in scandipwa.json (this means scandipwa/paypal-graphql):

    "require": {
        "scandipwa/paypal-graphql": "^2.0"
    },

But here again i get a similar error:

➜  pwa git:(master) composer install
Loading composer repositories with package information
Updating dependencies (including require-dev)
Your requirements could not be resolved to an installable set of packages.

  Problem 1
    - scandipwa/paypal-graphql 2.0.2 requires magento/magento2-base ^2.3.2 -> no matching package found.
    - scandipwa/paypal-graphql 2.0.1 requires magento/magento2-base ^2.3.2 -> no matching package found.
    - scandipwa/paypal-graphql 2.0.0 requires magento/magento2-base ^2.3.2 -> no matching package found.
    - Installation request for scandipwa/paypal-graphql ^2.0 -> satisfiable by scandipwa/paypal-graphql[2.0.0, 2.0.1, 2.0.2].

Potential causes:
 - A typo in the package name
 - The package is not available in a stable-enough version according to your minimum-stability setting
   see <https://getcomposer.org/doc/04-schema.md#minimum-stability> for more details.
 - It's a private package and you forgot to add a custom repository to find it

Read <https://getcomposer.org/doc/articles/troubleshooting.md> for further common problems.

In the meantime i will continue without the composer dependencies.

As i have something that seem to work, i will give up v2 to focus on v3 with its new middleware feature =)