Docker Image for development by allanbatista · Pull Request #62 · marvin-ai/marvin-python-toolbox

allanbatista · 2018-02-28T20:37:18Z

Was development two Docker images, one generic with marvin toolbox only and other to dev with spark and user included.

# expose ports 8000 and 8888 to random ports
$ docker run -it -P allanbatista/marvin-dev:0.0.2 /bin/bash -c "su marvin"

# expose ports 8000 and 8888
$ docker run -it -p 8000:8000 -p 8888:8888 allanbatista/marvin-dev:0.0.2 /bin/bash -c "su marvin"

# use toolbox env
$ workon marvin-toolbox-env

# create new engine
$ marvin engine-generate
$ workon NAME-engine-env
$ marvin notebook

codecov-io · 2018-02-28T20:49:06Z

Codecov Report

Merging #62 into master will increase coverage by 0.7%.
The diff coverage is n/a.

@@            Coverage Diff            @@
##           master      #62     +/-   ##
=========================================
+ Coverage   55.77%   56.47%   +0.7%     
=========================================
  Files          25       24      -1     
  Lines        2123     2130      +7     
  Branches      217      219      +2     
=========================================
+ Hits         1184     1203     +19     
+ Misses        924      910     -14     
- Partials       15       17      +2

Impacted Files	Coverage Δ
marvin_python_toolbox/common/utils.py	`100% <0%> (ø)`	⬆️
...thon_toolbox/engine_base/stubs/actions_pb2_grpc.py	`40% <0%> (ø)`	⬆️
marvin_python_toolbox/common/__init__.py
marvin_python_toolbox/common/data.py	`80.7% <0%> (+0.34%)`	⬆️
marvin_python_toolbox/management/engine.py	`54.59% <0%> (+1.01%)`	⬆️
...python_toolbox/engine_base/engine_base_training.py	`100% <0%> (+3.33%)`	⬆️
...n_python_toolbox/engine_base/engine_base_action.py	`70.42% <0%> (+8.45%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c3f4e1e...c6dd193. Read the comment docs.

oldfly · 2018-03-13T13:45:02Z

Hi @allanbatista, thanks for your PR!

I did some tests, before creating a new engine, if i try to run all unit tests with "marvin test" command, it triggers the IOError about file access permission and I got disconnected from docker immediately.

Am i missing something?

oldfly · 2018-03-14T16:36:39Z

Error log:

INTERNALERROR> Traceback (most recent call last):
INTERNALERROR> File "/opt/marvin/toolbox/.virtualenvs/marvin-toolbox-env/local/lib/python2.7/site-packages/_pytest/main.py", line 94, in wrap_session
INTERNALERROR> session.exitstatus = doit(config, session) or 0
INTERNALERROR> File "/opt/marvin/toolbox/.virtualenvs/marvin-toolbox-env/local/lib/python2.7/site-packages/_pytest/main.py", line 125, in _main
INTERNALERROR> config.hook.pytest_runtestloop(session=session)
INTERNALERROR> File "/opt/marvin/toolbox/.virtualenvs/marvin-toolbox-env/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 724, in call
INTERNALERROR> return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR> File "/opt/marvin/toolbox/.virtualenvs/marvin-toolbox-env/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 338, in _hookexec
INTERNALERROR> return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR> File "/opt/marvin/toolbox/.virtualenvs/marvin-toolbox-env/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 333, in
INTERNALERROR> _MultiCall(methods, kwargs, hook.spec_opts).execute()
INTERNALERROR> File "/opt/marvin/toolbox/.virtualenvs/marvin-toolbox-env/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 595, in execute
INTERNALERROR> return _wrapped_call(hook_impl.function(*args), self.execute)
INTERNALERROR> File "/opt/marvin/toolbox/.virtualenvs/marvin-toolbox-env/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 249, in _wrapped_call
INTERNALERROR> wrap_controller.send(call_outcome)
INTERNALERROR> File "/opt/marvin/toolbox/.virtualenvs/marvin-toolbox-env/local/lib/python2.7/site-packages/pytest_cov/plugin.py", line 235, in pytest_runtestloop
INTERNALERROR> self.cov_controller.finish()
INTERNALERROR> File "/opt/marvin/toolbox/.virtualenvs/marvin-toolbox-env/local/lib/python2.7/site-packages/pytest_cov/engine.py", line 115, in summary
INTERNALERROR> total = self.cov.html_report(ignore_errors=True, directory=self.cov_report['html'])
INTERNALERROR> File "/opt/marvin/toolbox/.virtualenvs/marvin-toolbox-env/local/lib/python2.7/site-packages/coverage/control.py", line 1095, in html_report
INTERNALERROR> return reporter.report(morfs)
INTERNALERROR> File "/opt/marvin/toolbox/.virtualenvs/marvin-toolbox-env/local/lib/python2.7/site-packages/coverage/html.py", line 139, in report
INTERNALERROR> self.report_files(self.html_file, morfs, self.config.html_dir)
INTERNALERROR> File "/opt/marvin/toolbox/.virtualenvs/marvin-toolbox-env/local/lib/python2.7/site-packages/coverage/report.py", line 91, in report_files
INTERNALERROR> report_fn(fr, self.coverage._analyze(fr))
INTERNALERROR> File "/opt/marvin/toolbox/.virtualenvs/marvin-toolbox-env/local/lib/python2.7/site-packages/coverage/html.py", line 286, in html_file
INTERNALERROR> write_html(html_path, html)
INTERNALERROR> File "/opt/marvin/toolbox/.virtualenvs/marvin-toolbox-env/local/lib/python2.7/site-packages/coverage/html.py", line 69, in write_html
INTERNALERROR> with open(fname, "wb") as fout:
INTERNALERROR> IOError: [Errno 13] Permission denied: u'coverage_report/marvin_python_toolbox_common_data_py.html'

allanbatista · 2018-03-15T20:45:24Z

Hi @oldfly.

I See. I will resolve soon.

allanbatista · 2018-03-16T20:23:16Z

@oldfly , fix that problem.

This happen because before complete build dev image, the process run marvin test, but they not create correct permission after.

$ docker run -it -p 8000:8000 -p 8888:8888 allanbatista/marvin-dev:0.0.2 /bin/bash -c "su marvin"
marvin@a2c851c558ea:/opt/marvin$ workon marvin-toolbox-env
(marvin-toolbox-env) marvin@a2c851c558ea:/opt/marvin/toolbox$ marvin test
py.test --cov marvin_python_toolbox --cov-report html --cov-report xml --cov-report term-missing tests
================================================================== test session starts ==================================================================
platform linux2 -- Python 2.7.12, pytest-2.9.2, py-1.5.2, pluggy-0.3.1
rootdir: /opt/marvin/toolbox, inifile: pytest.ini
plugins: testmon-0.9.9, flask-0.10.0, cov-2.5.1
collected 112 items 

tests/test_loader.py .
tests/common/test_config.py ............
tests/common/test_data.py .......
tests/common/test_data_source_provider.py ...
tests/common/test_http_client.py ..............
tests/common/test_profiling.py ...........
tests/common/test_utils.py .......................
tests/engine_base/test_engine_base_action.py .............
tests/engine_base/test_engine_base_data_handler.py ....
tests/engine_base/test_engine_base_prediction.py ....
tests/engine_base/test_engine_base_training.py .....
tests/management/test_engine.py ...
tests/management/test_notebook.py ..
tests/management/test_pkg.py ..........

---------- coverage: platform linux2, python 2.7.12-final-0 ----------
Name                                                            Stmts   Miss Branch BrPart  Cover   Missing
-----------------------------------------------------------------------------------------------------------
marvin_python_toolbox/__init__.py                                   6      0      0      0   100%
marvin_python_toolbox/_compatibility.py                             7      0      0      0   100%
marvin_python_toolbox/_logging.py                                  29      2      0      0    93%   85-86
marvin_python_toolbox/common/__init__.py                            2      0      0      0   100%
marvin_python_toolbox/common/config.py                             65      2     14      0    97%   52-53
marvin_python_toolbox/common/data.py                               56     19     14      2    61%   98-129, 68->70, 97->98
marvin_python_toolbox/common/data_source_provider.py               11      0      2      0   100%
marvin_python_toolbox/common/exceptions.py                          5      0      0      0   100%
marvin_python_toolbox/common/http_client.py                        91      0     12      0   100%
marvin_python_toolbox/common/profiling.py                          94      0     23      0   100%
marvin_python_toolbox/common/utils.py                             135      0     44      0   100%
marvin_python_toolbox/config.py                                    44     33     14      0    19%   37-55, 59-86
marvin_python_toolbox/decorators.py                                 9      9      0      0     0%   18-29
marvin_python_toolbox/engine_base/__init__.py                       5      0      0      0   100%
marvin_python_toolbox/engine_base/engine_base_action.py           142     51     26      3    63%   114-131, 139, 152, 155-160, 163-176, 179-182, 190, 193-198, 201-214, 217-220, 74->79, 136->140, 138->139
marvin_python_toolbox/engine_base/engine_base_data_handler.py      22      0      0      0   100%
marvin_python_toolbox/engine_base/engine_base_prediction.py        22      0      0      0   100%
marvin_python_toolbox/engine_base/engine_base_training.py          30      1      0      0    97%   69
marvin_python_toolbox/engine_base/stubs/__init__.py                 2      0      0      0   100%
marvin_python_toolbox/engine_base/stubs/actions_pb2.py            150     52      2      0    66%   485-495, 509-510, 516-517, 523-524, 529-548, 561-571, 585-586, 592-593, 599-600, 605-624, 638, 642, 646, 680-696, 705-721, 735, 739, 743, 777-793, 802-820
marvin_python_toolbox/engine_base/stubs/actions_pb2_grpc.py        40     24      0      0    40%   17-27, 41-42, 48-49, 55-56, 61-80, 93-103, 117-118, 124-125, 131-132, 137-156
marvin_python_toolbox/loader.py                                     7      0      0      0   100%
marvin_python_toolbox/manage.py                                     1      0      0      0   100%
marvin_python_toolbox/management/__init__.py                       55     38     18      0    23%   64-135
marvin_python_toolbox/management/engine.py                        392    173    112      9    52%   45, 67, 142-143, 207-211, 218-221, 231, 233, 235, 237, 239, 251-272, 294-335, 358-363, 376-451, 458-461, 465-472, 476-505, 509-518, 522-533, 537-543, 547-552, 578, 607-609, 625-628, 648-670, 681-697, 216->218, 230->231, 232->233, 234->235, 236->237, 238->239, 594->599, 596->595, 612->616
marvin_python_toolbox/management/hive.py                          392    315     50      0    17%   38, 44-56, 65-79, 106-146, 150-156, 166-203, 207-223, 226-249, 252-282, 285, 288-291, 297-442, 446, 450, 454, 458, 461, 464-491, 494-497, 500-509, 512-519, 522, 525, 528, 531, 534-537, 540, 546-555, 562-567, 570-571, 574-580, 583-589, 592-598, 601-604, 607-608, 611-612, 615-619, 622-623, 626-627, 630-636, 639-642, 646-674, 677-681
marvin_python_toolbox/management/notebook.py                       18      2      0      0    89%   27, 36
marvin_python_toolbox/management/pkg.py                           297    197     96      0    31%   38, 44, 50, 56-88, 94-135, 148-173, 179-182, 190-235, 241-245, 251-253, 281-294, 298-312, 319-364
marvin_python_toolbox/management/test.py                           80     52     22      0    27%   33, 43-69, 76-90, 101-131, 137-141
-----------------------------------------------------------------------------------------------------------
TOTAL                                                            2209    970    449     14    53%
Coverage HTML written to dir coverage_report
Coverage XML written to file coverage.xml


============================================================== 112 passed in 14.87 seconds ============================================================

Please, do not forget to update image before test again.

docker pull allanbatista/marvin-dev:0.0.2

oldfly · 2018-03-23T15:22:40Z

Hi @allanbatista, Permission fix checked, thanks!

And after marvin test, the docker container is halted automatically, please check this behavior too!

takabayashi · 2018-05-03T16:32:21Z

Hi @allanbatista lets finish and merge this code? To do that you must fix the last bug reported by @oldfly ...

allanbatista added 2 commits February 27, 2018 13:35

Init create Dockerfile-base

dc24e5a

create a docker image do developer

7a5a631

fix docker user permission

c6dd193

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Docker Image for development#62

Docker Image for development#62
allanbatista wants to merge 3 commits intomarvin-ai:masterfrom
allanbatista:docker-image-dev

allanbatista commented Feb 28, 2018

Uh oh!

codecov-io commented Feb 28, 2018 •

edited

Loading

Uh oh!

oldfly commented Mar 13, 2018 •

edited

Loading

Uh oh!

oldfly commented Mar 14, 2018 •

edited

Loading

Uh oh!

allanbatista commented Mar 15, 2018 •

edited

Loading

Uh oh!

allanbatista commented Mar 16, 2018

Uh oh!

oldfly commented Mar 23, 2018

Uh oh!

takabayashi commented May 3, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

allanbatista commented Feb 28, 2018

Uh oh!

codecov-io commented Feb 28, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

oldfly commented Mar 13, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

oldfly commented Mar 14, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

allanbatista commented Mar 15, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

allanbatista commented Mar 16, 2018

Uh oh!

oldfly commented Mar 23, 2018

Uh oh!

takabayashi commented May 3, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codecov-io commented Feb 28, 2018 •

edited

Loading

oldfly commented Mar 13, 2018 •

edited

Loading

oldfly commented Mar 14, 2018 •

edited

Loading

allanbatista commented Mar 15, 2018 •

edited

Loading