Set auto_envvar_prefix to STREAMLIT #477

kinghuang · 2019-10-20T23:24:57Z

Issue: #476

Description: Set auto_envvar_prefix to STREAMLIT to accept parameters from environment variables in addition to regular CLI arguments. This is a Click option that will automatically get values from env vars if available.

Contribution License Agreement

By submiting this pull request you agree that all contributions to this project are made under the Apache 2.0 license.

tvst · 2019-10-22T08:57:01Z

Awesome!

I'm having trouble testing this, though. How do I set the --server.port CLI argument via an environment variable?

The problem I'm running into is that STREAMLIT_SERVER.PORT is not a valid env var name...

kinghuang · 2019-10-23T15:13:59Z

I'm looking into this.


        Set auto_envvar_prefix to STREAMLIT

Set auto_envvar_prefix to STREAMLIT to accept parameters from environment variables in addition to regular CLI arguments. This is a Click option that will automatically get values from env vars if available.

kinghuang · 2019-10-23T16:00:14Z

@tvst I've moved auto_envvar_prefix to the context_settings on the main command group so that its applied whether the command is invoked through cli.py or via the streamlit entrypoint.

The format of the environment variables that Click generates is PREFIX_COMMAND_VARIABLE. So, for the --server.port option on the run sub-commmand, the corresponding environment variable will be STREAMLIT_RUN_SERVER_PORT. Click converts periods to underscores automatically.

root@104bd402c055:/project# streamlit run sa/gui.py

  You can now view your Streamlit app in your browser.

  Network URL: http://172.19.0.2:8501
  External URL: http://209.89.162.170:8501

^C  Stopping...
root@104bd402c055:/project# export STREAMLIT_RUN_SERVER_PORT=5678
root@104bd402c055:/project# streamlit run sa/gui.py

  You can now view your Streamlit app in your browser.

  Network URL: http://172.19.0.2:5678
  External URL: http://209.89.162.170:5678

^C  Stopping...

If you temporarily set show_envvar=True on the dynamically added options in configuration_options(func), the help text will show the corresponding environment name for each option. Here's an excerpt for --server.port.

  --server.port INTEGER           The port where the server will listen for
                                  client and browser connections.
                                  
                                  Default: 8501  [env var: STREAMLIT_RUN_SERVER_PORT]

I noticed that you attempted to set STREAMLIT_SERVER.PORT. After playing with this some more, I'm starting to think that environment variables as options should be limited to just the run command (run_main). That way there isn't a RUN in the middle of the environment name, making STREAMLIT_SERVER_PORT the corresponding environment name for the --server.port option, for example. That also means that Click won't automatically read environment variable values for options on the other commands.

With the prefix on run_main instead of the root main:

  --server.port INTEGER           The port where the server will listen for
                                  client and browser connections.
                                  
                                  Default: 8501  [env var: STREAMLIT_SERVER_PORT]

kinghuang · 2019-10-23T16:21:18Z

Another option is to leave auto_envvar_prefix at the root, but then override the options added by configurator_options(func) with custom envvar names. They could be set to something like STREAMLIT_CONFIG_SERVER_PORT for the --server.port option.

This overlaps with the config command, but I think that's acceptable given it has no real options of its own, and it's about the options that it's showing.

Another thing that might be useful is assigning an envvar to the file_or_url argument on the run command. That would make it possible to specify the file or url to run via an environment variable, too. This would be handy from a container perspective.

root@1313373f8861:/project# export STREAMLIT_RUN_FILE_OR_URL=sa/gui.py
root@1313373f8861:/project# streamlit run

  You can now view your Streamlit app in your browser.

  Network URL: http://172.19.0.2:8501
  External URL: http://209.89.162.170:8501


        Customize envvar names for config options

Override the environment variable names for the options dynamically added by configurator_options to be STREAMLIT_CONFIG_{option}. So, for --server.port, the corresponding envvar is STREAMLIT_CONFIG_SERVER_PORT instead of STREAMLIT_RUN_SERVER_PORT.


        Set envvar for file_or_url argument

Set the environment variable for the file_or_url argument on the run command to STREAMLIT_RUN_FILE_OR_URL. Click does not do automatic envvars for arguments. This makes it possible to specify the file or url to run using an environment variable.

kinghuang · 2019-10-23T16:29:01Z

I've pushed commits to override the envvar names for the config options and add an envvar for the file_or_url argument. Here's an example of their use.

root@1313373f8861:/project# export STREAMLIT_CONFIG_SERVER_PORT=2468
root@1313373f8861:/project# export STREAMLIT_RUN_FILE_OR_URL=sa/gui.py
root@1313373f8861:/project# streamlit run

  You can now view your Streamlit app in your browser.

  Network URL: http://172.19.0.2:2468
  External URL: http://209.89.162.170:2468

treuille · 2019-10-24T04:49:28Z

Awesome PR @kinghuang!

Two questions (@tvst should be the decider):

STREAMLIT_RUN_FILE_OR_URL sounds weird and limiting. How about STREAMLIT_RUN_TARGET?
Why do we need this at all? Can’t we just do the following?

$ export STREAMLIT_RUN_TARGET=sa/gui.py
$ streamlit run ${STREAMLIT_RUN_TARGET}

tvst

Just tried this and it's really amazing! 💯

Can you also add this to docs/cli.md line 63:

Shows all config options available for Streamlit, including their current
values. You can set these options in four different ways:

- **In a global config file at `~/.streamlit/config.toml`.** For instance:
  ```toml
  [server]
  port = 80
  ```

- **In a per-project config file at `$CWD/.streamlit/config.toml`,** where
  `$CWD` is the folder you're running Streamlit from.

- **Through `STREAMLIT_CONFIG_*` environment variables,** such as:
  ```bash
  $ export STREAMLIT_CONFIG_SERVER_PORT=80
  ```

- **As flags in the command line** when running `streamlit run`. For example:
  ```bash
  $ streamlit run your_script.py --server.port 80
  ```

And while you're at it, do you mind fixing line 26? 😉

.. note::
  When passing your script some custom arguments, **they must be passed after a
  "--"** (double dash). Otherwise the arguments get interpreted as arguments to
  Streamlit itself.

tvst · 2019-10-24T04:52:58Z

lib/streamlit/cli.py

@@ -147,6 +147,7 @@ def configurator_options(func):
            parsed_parameter["param"],
            help=parsed_parameter["description"],
            type=parsed_parameter["type"],
+            envvar='STREAMLIT_CONFIG_{}'.format(parsed_parameter["param"].upper()),


Make the suffix use snake case too. We have a function called to_snake_case() in case_converters.py that should do the trick.

In which case this line will get really long and nasty, so you could pull this logic into the _convert_config_option_to_click_option function.

Even better if we could change line 145 to just

config_option = click.option(**parsed_parameter)

Sure, I'll take a look at all these tomorrow!

I've moved the logic into _convert_config_option_to_click_option(). But, I'm not so sure about applying to_snake_case.

For an option like server.baseUrlPath, we have:

STREAMLIT_CONFIG_SERVER_BASE_URL_PATH (with snakecase)
STREAMLIT_CONFIG_SERVER_BASEURLPATH (without snakecase)

I think the without snakecase version matches the underlying config name better. With the snakecase version, I wouldn't know if it's supposed to match a config named server.base.url.path, server.baseUrl.path, serverBaseUrlPath, etc.

The plan is to have only a two-level hierarchy for config options, so there shouldn't be any ambiguity so I think STREAMLIT_CONFIG_SERVER_BASE_URL_PATH is better.

The non-snakecase version has already been merged. But, here's how some other projects do config to environment variable mappings, for comparison. Overall, I feel that converting camelCase to straight uppercase without snake-casing is the better choice because it matches more closely with other apps.

Click's Default Behaviour

Turning on automatic environment variables in Click converts option names to uppercase, without snake casing.

--server.baseUrlPath == {PREFIX}_SERVER_BASEURLPATH

Traefik

Traefik's configs have camelCase names, and are mapped to CLI options in lowercase. The corresponding environment variables are not snake-cased.

[accessLog] filePath == --accesslog.filepath == TRAEFIK_ACCESSLOG_FILEPATH

Kafka (using Confluent Platform images)

Kafka's configs are all lowercase, and tend to have each word separated by a period (as opposed to camelCasing or snake_casing). In the Confluent Platform image for Kafka, the corresponding environment variables are only uppercased.

auto.create.topics.enable == KAFKA_AUTO_CREATE_TOPICS_ENABLE

Grafana

Grafana's configs are snake cased, which are preserved in environment variables.

[server] http_port == GF_SERVER_HTTP_PORT

Ooops, I think I merged this too soon. No problem, we'll address it in another PR.

kinghuang · 2019-10-24T05:07:54Z

For @treuille's questions:

STREAMLIT_RUN_TARGET sounds good to me, too. It might also help with PR #79. Maybe the underlying argument should be renamed from file_or_url to target?
The downside to streamlit run ${STREAMLIT_RUN_TARGET} is that it prevents containers from writing that in their entrypoint or command in exec form. There's no shell in the exec form to expand variables. While writing them in shell form is not the end of the world, the exec form is preferred.


        Move envvar formatting to _convert_config_option_to_click_option

Move the logic for constructing an environment variable name out of configuration_options() to _convert_config_option_to_click_option(), where the other Click options are prepped.


        Update CLI docs for config options

@tvst

Add the copy from @tvst explaining the now four ways Streamlit can be configured.


        Clarify that -- is needed for custom arguments

@tvst

Add the updated copy from @tvst about delineating custom script arguments with --.

tvst · 2019-10-24T18:23:11Z

STREAMLIT_RUN_TARGET sounds good to me, too. It might also help with PR #79. Maybe the underlying argument should be renamed from file_or_url to target?

👍
"Target" is much better than "file_or_url"


        Rename file_or_url to target

tvst · 2019-10-25T00:11:44Z

Ok, I just changed it to target in the PR 😃

kinghuang · 2019-10-25T01:17:56Z

Awesome. I couldn't quite squeeze in that change in between meetings, today. :D

treuille · 2019-10-25T23:02:14Z

The downside to streamlit run ${STREAMLIT_RUN_TARGET} is that it prevents containers from writing that in their [entrypoint]

Ah. That's compelling. Still I feel like streamlit run is kinda weird, but I don't feel strongly. I can go either way. Any thoughts @tvst ?

treuille · 2019-10-25T23:02:21Z

Btw, this is really exciting work, @kinghuang !


        Merge branch 'develop' into tim/StreamlitException

* develop: Use custom context manager for temporary file in cli.py (streamlit#489) Update pull_request_template.md Update pull_request_template.md Handle lack of trailing slash (streamlit#528) Set auto_envvar_prefix to STREAMLIT (streamlit#477) Strip slashes in path components (streamlit#523) Create doc_improvement.md ESlint: allow @ts-ignore (streamlit#499) Update Pipfile (streamlit#505) Release 0.49.0 (streamlit#509) Removing package json (streamlit#501) Feature/input number (streamlit#416) ESLint warnings are fatal in Circle (streamlit#492) Doc updates (streamlit#286)

dijitali · 2020-01-15T15:53:31Z

Thanks for this PR @kinghuang , it's exactly what I'm after but I'm just a bit confused on the correct usage.

The v0.50.1 release notes state:

You can now set config options using environment variables. For example, export STREAMLIT_SERVER_PORT=9876.

But the docs state:

Through STREAMLIT_CONFIG_* environment variables, such as:
$ export STREAMLIT_CONFIG_SERVER_PORT=80

I've tried setting both STREAMLIT_SERVER_BASEURLPATH and STREAMLIT_CONFIG_SERVER_BASE_URL_PATH and neither seem to take effect (setting it in the config.toml does work).

kinghuang · 2020-01-15T18:51:26Z

@dijitali Ah, the docs are incorrect! The CONFIG part was intentionally dropped from the final environment variables as shown in the release notes, but I missed the corresponding change in the docs. I'll fix that up.

For the server.baseUrlPath option, the corresponding environment variable is STREAMLIT_SERVER_BASE_URL_PATH.

root@1d1e0b2f6b03:/# export STREAMLIT_SERVER_BASE_URL_PATH=/abc
root@1d1e0b2f6b03:/# streamlit run st.py

  You can now view your Streamlit app in your browser.

  Network URL: http://172.17.0.3:8501/abc
  External URL: http://209.89.162.170:8501/abc

This is also shown in the help text for the run command.

root@1d1e0b2f6b03:/# streamlit run --help
Usage: streamlit run [OPTIONS] TARGET [ARGS]...

Options:
  --server.baseUrlPath TEXT       The base path for the URL where Streamlit
                                  should be served from.  [env var:
                                  STREAMLIT_SERVER_BASE_URL_PATH]

dijitali · 2020-01-16T09:20:29Z

Perfect, many thanks. Wasn't aware the --help command had all those details.

kinghuang requested a review from streamlit/eng as a code owner Oct 20, 2019

kinghuang changed the title ~~WIP: Set auto_envvar_prefix to STREAMLIT~~ Set auto_envvar_prefix to STREAMLIT Oct 21, 2019

tvst self-requested a review Oct 22, 2019

kinghuang force-pushed the kinghuang:476-auto-envvar branch from 8713b58 to 2f7c9e0 Oct 23, 2019

Set auto_envvar_prefix to STREAMLIT

Loading status checks…

1ed7204

Set auto_envvar_prefix to STREAMLIT to accept parameters from environment variables in addition to regular CLI arguments. This is a Click option that will automatically get values from env vars if available.

kinghuang force-pushed the kinghuang:476-auto-envvar branch from 2f7c9e0 to 1ed7204 Oct 23, 2019

kinghuang added 2 commits Oct 23, 2019

tvst approved these changes Oct 24, 2019

View changes

kinghuang added 3 commits Oct 24, 2019

Move envvar formatting to _convert_config_option_to_click_option

821be1f

Move the logic for constructing an environment variable name out of configuration_options() to _convert_config_option_to_click_option(), where the other Click options are prepped.

Update CLI docs for config options

56fa375

Add the copy from @tvst explaining the now four ways Streamlit can be configured.

Clarify that -- is needed for custom arguments

Loading status checks…

edb1992

Add the updated copy from @tvst about delineating custom script arguments with --.

Rename file_or_url to target

Verified

This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.

GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits

Loading status checks…

abcda35

kinghuang deleted the kinghuang:476-auto-envvar branch Oct 25, 2019

kinghuang mentioned this pull request Oct 25, 2019

Adding `run -m` option to run modules #79

Closed

kinghuang mentioned this pull request Oct 26, 2019

Allow environment variables to be used to set CLI options #476

Closed

dnuske mentioned this pull request Oct 28, 2019

Extends capability to send config keys by param to "streamlit hello" command #527

Merged

kinghuang mentioned this pull request Nov 9, 2019

Add option to run modules #662

Open

kinghuang mentioned this pull request Jan 15, 2020

Fix docs for setting config options through environment variables #976

Open

streamlit / streamlit

Join GitHub today

Set auto_envvar_prefix to STREAMLIT #477

Set auto_envvar_prefix to STREAMLIT #477

Conversation

kinghuang commented Oct 20, 2019

This comment has been minimized.

tvst commented Oct 22, 2019

This comment has been minimized.

kinghuang commented Oct 23, 2019

This comment has been minimized.

kinghuang commented Oct 23, 2019 • edited

This comment has been minimized.

kinghuang commented Oct 23, 2019

This comment has been minimized.

kinghuang commented Oct 23, 2019

This comment has been minimized.

treuille commented Oct 24, 2019

tvst left a comment

This comment has been minimized.

tvst Oct 24, 2019

This comment has been minimized.

kinghuang Oct 24, 2019

This comment has been minimized.

kinghuang Oct 24, 2019

This comment has been minimized.

treuille Oct 25, 2019

This comment has been minimized.

kinghuang Oct 26, 2019 • edited

This comment has been minimized.

tvst Oct 28, 2019

This comment has been minimized.

kinghuang commented Oct 24, 2019

This comment has been minimized.

tvst commented Oct 24, 2019

This comment has been minimized.

tvst commented Oct 25, 2019

This comment has been minimized.

kinghuang commented Oct 25, 2019

This comment has been minimized.

treuille commented Oct 25, 2019

This comment has been minimized.

treuille commented Oct 25, 2019

This comment has been minimized.

dijitali commented Jan 15, 2020

This comment has been minimized.

kinghuang commented Jan 15, 2020

This comment has been minimized.

dijitali commented Jan 16, 2020

kinghuang commented Oct 23, 2019 •

edited

kinghuang Oct 26, 2019 •

edited