As implemented, module-scoped constants may not work properly

I noticed that your schema's constants get translated to naked values, e.g. capnpc-ts/test/integration/test.capnp.ts#L2110-L2145. Unfortunately, ES6 and Node's module resolution don't guarantee that imports are available at module scope. When you go to wrap an imported struct around some data for a module-scoped constant, you may discover that the imported struct is not yet available. Translating someConstant to getSomeConstant() generalizes to module scope without any problems, but I'm curious if you've got some other workaround.

I might be missing something here, but isn't this only a problem when you
have circular imports?
That's a thing to be avoided at all costs, and I certainly remember having
to go out of my way to prevent it within this library.
A minimal example of how this might break would help.
On Wed, Dec 19, 2018, 6:53 PM Tim Popham<EMAIL_ADDRESS>wrote:

I noticed that your schema's constants get translated to naked values,
e.g. capnpc-ts/test/integration/test.capnp.ts#L2110-L2145
https://github.com/jdiaz5513/capnp-ts/blob/524b6bde7d81037de586e2905d1d40955bbe22b0/packages/capnpc-ts/test/integration/test.capnp.ts#L2110-L2145.
Unfortunately, ES6 and Node's module resolution don't guarantee that
imports are available at module scope. When you go to wrap an imported
struct around some data for a module-scoped constant, you may discover that
the imported struct is not yet available. Translating someConstant to
getSomeConstant() generalizes to module scope without any problems, but
I'm curious if you've got some other workaround.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/jdiaz5513/capnp-ts/issues/113, or mute the thread
https://github.com/notifications/unsubscribe-auth/ACn_Z5fb8ldqWfEm0Opg4q3KmpVhk4Zyks5u6vuZgaJpZM4ZbZkH
.

The problem will pop up in generated code. The Cap'n Proto schema language admits circular imports, so your generated code will encounter circular imports like the following:
// file1.capnp
constant c1 :import "file2.capnp".F2 = ();
struct F1 {}

// file2.capnp
constant c2 :import "file1.capnp".F1 = ();
struct F2 {}

If you're exposing value assignments as scoped constants, then a constant at a schema's file scope translates to an assignment at module scope, e.g.
//file1.capnp.ts
const c1 = new F2...;

//file2.capnp.ts
const c2 = new F1...;

When the module initializes, the F1 and F2 are not necessarily available for the news.
I should monkey around with the code anyway; I'll submit a PR that amends packages/capnpc-ts/test/integration/import-bar.capnp and packages/capnpc-ts/test/integration/import-foo.capnp to demonstrate.

Your struct-typed constants currently call capnp.rawPointer to yield a Pointer instance, so circular imports will work fine as currently implemented. If you intend to ever construct the actual struct types, however, then my proposed problem will arise. I was only able to look at struct-scoped constants because module-scoped constants did not get translated.

I've added a hypothetical export of the module-scoped constant to #116. Running tests as written clobbered the hand coded additions, so I've altered gulp's test:capnpc-js task to stop the task from overwriting the hand-coded additions. Running gulp test:capnpc-js, note the error: import _foo_capnp_1.Foo is not a constructor. It chokes on the constructor before it even tries to work with the (null as any)s. Toss in a if (Foo===undefined) { throw new Error("PEEP"); } just before the const FOO =... line, and you'll get a "PEEP".


GEODE-6998: Fix for NPE at index update due to GII

Thank you for submitting a contribution to Apache Geode.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:

[*] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message?

[*] Has your PR been rebased against the latest commit within the target branch (typically develop)?

[*] Is your initial contribution a single, squashed commit?

[*] Does gradlew build run cleanly?

[*] Have you written or updated unit tests to verify your changes?

[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?

Note:
Please ensure that once the PR is submitted, check Concourse for build issues and
submit an update to your PR as soon as possible. If you need help, please send an
email to<EMAIL_ADDRESS>

I modified PartitionedRegionCompactRangeIndexDUnitTest locally to see if I could repro the issue.  It looks like this occurs if the current value in the cache is already a tombstone and the new value also happens to be a tombstone.  I think this change should be fine

@agingade What do you think about this change?  In particular, how do you feel about no-op'ing instead of crawling the index.

@jhuynh1 @mivanac
Jason, you are right no-oping is the right way to go here. We are getting into this situation, due to single GII processing thread updating index for batch of entries. And processing the entry that is removed. Since we don't store any removed-entries (TOMBSTONEs) in index, we could just return if oldKeyValue returns null.
// We know the old key so let's just remove mapping from the old key
if (oldKeyValue != null) {
if (oldKeyValue.get() == null) {
return;
}
indexStore.removeMapping(oldKeyValue.get().getOldKey(), entry);
}
We should add few DUnit test, reproducing the issue.

Thanks for comments. Code is updated. Do you have any proposals for DUnit test.

DUnit test added.

I'm currently working a dunit test but it might take another day to iron out.  If you want, I can create a new PR when it is done and reference this ticket number.

Thanks for help with DUnit test.

Could we merge this PR to develop now, and additionally add test, so this PR is included in 1.10 release.


error QNetworkReply::ProtocolInvalidOperationError

Hi, first of all, excellent project.
I ran the application (also compiled it from source to be sure) and got the output below. Any idea what I'm doing wrong? I think I have the correct user and project token. It keeps showing me the spinning wheel with an interrupt button (that doesn't do anything) and then goes into some infinite loop saying write componets! write elements
My figma project is a very simple project with some rectangles with gradient color and some colored text.
Thanks!
qrc:/QtCheckBox.qml:15:17: QML Rectangle: The current style does not support customization of this control (property: "background" item: QQuickRectangle(0x12b9e93dfb0, parent=0x0, geometry=0,0 0x0)). Please customize a non-native style (such as Basic, Fusion, Material, etc). For more information, see: https://doc.qt.io/qt-6/qtquickcontrols2-customize.html#customization-reference
qml: Save 1 1
qml: Frestore 1 1
write componets!
write elements
doRequestRendering "1:3" "RENDERING"  error QNetworkReply::ProtocolInvalidOperationError
qml: too many requests
qrc:/qt-project.org/imports/QtQuick/Dialogs/quickimpl/qml/MessageDialog.qml:120:25: QML Rectangle: The current style does not support customization of this control (property: "background" item: QQuickRectangle(0x12ba1e4c9d0, parent=0x0, geometry=0,0 0x0)). Please customize a non-native style (such as BaTimeout "1:4_timeout", etc). For more information,
Timeout "1:3_timeout"io/qt-6/qtquickcontrols2-customize.html#customization-reference
write componets!
write elements
doRetrieveImage "RENDERING" "1:3" false QUrl("")
qml: Rendering 1:3 Url not valid ""
qrc:/qt-project.org/imports/QtQuick/Dialogs/quickimpl/qml/MessageDialog.qml:120:25: QML Rectangle: The current style does not support customization of this control (property: "background" item: QQuickRectangle(0x12ba1e4e6b0, parent=0x0, geometry=0,0 0x0)). Please customize a non-native style (such as Basic, Fusion, Material, etc). For more information, see: https://doc.qt.io/qt-6/qtquickcontrols2-customize.html#customization-reference
doRetrieveImage "RENDERING" "1:4" false QUrl("")
qml: Rendering 1:4 Url not valid ""
write componets!
write elements
write componets!
write elements
<ad infinitum>

Have you just retried later? Figma server have strange issues time to time, maybe just busy, and then just try again later helps. Especially any rendering request is a bit problematic as then Figma.com does the graphics rendering (to PNG) for you, that is relatively hard operation vs. just post JSON - therefore it has tendency to fail. See FigmaQML settings and avoid ticking any options that says rendering.  I have to think alternatives for rendering, as that is intially implemented for testing

Hi, I've been trying on various different times now but unfortunately still the same. Also, if I click the settings button, all options are switched off, including all rendering options. Anything I can do from my end to help debug this? Enable some debug traces or anything?

If you build in QtCreator and run - Debug mode uses quit lot of asserts, so if we are lucky it crash and tell where is the problem, but I doubt with this case  - there is some debug console messages, but no specific logging.  What is your OS? I have not tried with Windows lately.

WASM version's networking get broken when I compiled it on 6.6.2 - Qt is investigating. Im waiting 6.7  - but prepared to fall back to 6.4

WASM version's networking get broken when I compiled it on 6.6.2 - Qt is investigating. Im waiting 6.7 - but prepared to fall back to 6.4

Hi, Is there a way to solve this problem? simple 1 rect design is ok, but more complex design also freeze with "RENDERING" QNetworkReply::ProtocolInvalidOperationError error.

Hi, that issue applies only an WASM version. Can you use the native built?
regards,
Markus
On Tue 18. Jun 2024 at 16.29, teatralaik @.***> wrote:

WASM version's networking get broken when I compiled it on 6.6.2 - Qt is
investigating. Im waiting 6.7 - but prepared to fall back to 6.4
Hi, Is there a way to solve this problem? simple 1 rect design is ok, but
more complex design also freeze with "RENDERING"
QNetworkReply::ProtocolInvalidOperationError error.
—
Reply to this email directly, view it on GitHub
https://github.com/mmertama/FigmaQML/issues/16#issuecomment-2176106900,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AB3ZQDOYFKHJ5P7N6WWURBLZIAY3PAVCNFSM6AAAAABJQBQNHOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNZWGEYDMOJQGA
.
You are receiving this because you commented.Message ID:
@.***>

Excuse me, but i don't understand what is "native" build. I download qt, and install it, then i clone source, build it by cmake and run in visual studio 22. All of this in Windows, not inside VM

I have this exact same problem. I am using Qt 6.7. Using 3.1 version of this build...
Infinity

I have this exact same problem. I am using Qt 6.7. Using 3.1 version of this build... Infinity

I have this exact same problem. I am using Qt 6.7. Using 3.1 version of this build... Infinity

It seems there are problems with Two methods in the figmaget.cpp, namely: doRequestRendering and doRetrieveImage.
The resolution for now would be to remove the images from the Figma design file and uncheck the Render view in the settings  of the compiled app.

Generating images is pretty heavy operation and likely Figma server throttles image rendering operations.

Generating images is pretty heavy operation and likely Figma server throttles image rendering operations.

hmm... That's good to know. So, there isn't any problem with doRetrieveImage and it's on Figma's side?

It think that is problem in Figma side, but I have look their API closer if there are something new to handle these issues

Sorry, I have missed some relative fixes from main. I just added them - it would be greatly helpful if your take a latest version.  If that wont help, I will look further fixes


Prompt and cleaning updates

Just realized this hasn't been pushed to main.

Added query cleaning post-generation to fix float divisions and spaces between >= and <=
Separate join_hints as a new field that can be added/removed within prompt template
Added join_hints to prompt_cot_postgres
Added pruned_join_hints field (removing obvious join columns with the same name). This is however only currently enabled for when columns_to_keep == 0)
Took the chance to modify one SQL to follow instructions that requests for percentage value instead of rate.

Yea for a while I was actually doubting myself about whether we had these changes made 😅


Pandas read_csv(squeeze='True') to enable koalas Series and not Dataframe

Would be extremely useful to directly read in a koalas.Series from read_csv and not have to convert Dataframe to Series (which also does not exist in Koalas).
Thank you!

@ueshin  I am working on this one.


jaeger agent can't connect to collector

jaeger-agent deployment command:
docker run  -d  --name agent   --rm   -p5775:5775/udp  -p6831:6831/udp   -p6832:6832/udp  -p5778:5778/tcp jaegertracing/jaeger-agent  /go/bin/agent-linux   --collector.host-port=localhost:14267
jaeger-collector deployment command:
docker run  -d --rm  -p16686:16686/tcp  jaegertracing/jaeger-query  /go/bin/query-linux  --es.server-urls=http://es-server:9200 --span-storage.type=elasticsearch  --query.static-files=/go/jaeger-ui/
logs of jaeger-agent container

The firewalld is inactive and SELINUX is disabled.

What was the issue? I'd like to get this documented for future users who run into this.

@black-adder  OK, i run jaeger-agent and jaeger-collector in the same machine, and  tell jaeger-agent  the host and port of jaeger-collector is localhost:14267. Because jaeger-agent run in docker container,  there no process listen on 14267 in the same container, so jaeger-agent cant' connect to jaeger-collector. It works when i tell jaeger-agent the machine host (not the localhost).

Hi,
I need  to run jaeger-agent, jaeger-collector and jaeger-ui in individual machines.
I tried to do this using the following commands, which is not success.
Please let me know where I am doing wrong and how to achieve this.
So, that I should send logs to collector in another machine and those should be read by jaeger-query in different machine.
Below are the commands
Jaeger Query / ui (UI Machine)
sudo docker run --rm -p16686:16686/tcp jaegertracing/jaeger-query /go/bin/query-linux --es.server-urls=http://IP-of-collector:9200 --span-storage.type=elasticsearch --query.static-files=/go/jaeger-ui/
Jeager Collector (Collector Machine)
sudo docker run --rm  jaegertracing/jaeger-collector /go/bin/collector-linux --es.server-urls=http://IP-of-collector:9200 --span-storage.type=elasticsearch --log-level=error
Jaeger Agent (Agent Machine)
sudo docker run --name agent --rm -p5775:5775/udp -p6831:6831/udp -p6832:6832/udp -p5778:5778/tcp jaegertracing/jaeger-agent /go/bin/agent-linux --collector.host-port=IP-of- collector:14267
Please let me know where I am wrong and how to fix this.

sudo docker run --rm jaegertracing/jaeger-collector /go/bin/collector-linux --es.server-urls=http://IP-of-collector:9200 --span-storage.type=elasticsearch --log-level=error

Your collector is not exposing any ports. You'll need -p14267:14267 -p14268:14268 -p9411:9411 as well.

@jpkrohling  thanks for the reply
I tried exposing the ports in the host.
I got the below output
From Collector
docker run --rm --env SPAN_STORAGE_TYPE=elasticsearch -p14267:14267/tcp -p14268:14268 -p9411:9411 jaegertracing/jaeger-collector /go/bin/collector-linux  --es.server-urls=http://es-ip:9200 --log-level=debug
{"level":"info","ts":1529300740.434398,"caller":"healthcheck/handler.go:99","msg":"Health Check server started","http-port":14269,"status":"unavailable"}
{"level":"info","ts":1529300740.6546476,"caller":"static/strategy_store.go:77","msg":"No sampling strategies provided, using defaults"}
{"level":"info","ts":1529300740.6550906,"caller":"collector/main.go:141","msg":"Registering metrics handler with HTTP server","route":"/metrics"}
{"level":"info","ts":1529300740.655314,"caller":"collector/main.go:149","msg":"Starting Jaeger Collector HTTP server","http-port":14268}
{"level":"info","ts":1529300740.6554735,"caller":"healthcheck/handler.go:133","msg":"Health Check state change","status":"ready"}
Form Agent
{"level":"info","ts":1529300741.6044064,"caller":"peerlistmgr/peer_list_mgr.go:166","msg":"Trying to connect to peer","host:port":"<IP_ADDRESS>:14267"}
{"level":"info","ts":1529300741.6142902,"caller":"peerlistmgr/peer_list_mgr.go:176","msg":"Connected to peer","host:port":"[::]:14267"}
It is showing that both Collector and Agent are connected to each other.
But once i run run the application, I can't see any logs in Jaeger UI , It is not creating the service also.
How to fix this?
How to find collector are collecting the logs.

The first step is to make sure your instrumented application is reporting logs. Depending on which client you are using and how you are instantiating it, it could be a matter of just setting the env var JAEGER_REPORTER_LOG_SPANS to true:
https://github.com/jaegertracing/jaeger-client-java/tree/master/jaeger-core#configuration-via-environment
https://github.com/jaegertracing/jaeger-client-go#environment-variables

There's nothing about Jaeger/OpenTracing on the logs you've posted. It's hard to tell what's going on without seeing Jaeger's logs and not knowing your code...

Hi,
I am facing similiar issues while running components in same machine.
sudo docker run -it --rm --env SPAN_STORAGE_TYPE=memory -p14267:14267/tcp -p14268:14268/tcp -p9411:9411/tcp jaegertracing/jaeger-collector /go/bin/collector-linux --log-level debug
{"level":"info","ts":1529393916.8745806,"caller":"collector/main.go:149","msg":"Starting Jaeger Collector HTTP server","http-port":14268}
{"level":"info","ts":1529393916.8746583,"caller":"healthcheck/handler.go:133","msg":"Health Check state change","status":"ready"}
sudo docker run   --rm   -p5775:5775/udp   -p6831:6831/udp   -p6832:6832/udp   -p5778:5778/tcp   jaegertracing/jaeger-agent  /go/bin/agent-linux --collector.host-port=<IP_ADDRESS>:14267
{"level":"error","ts":1529394232.9935606,"caller":"peerlistmgr/peer_list_mgr.go:171","msg":"Unable to connect","host:port":"<IP_ADDRESS>:14267","connCheckTimeout":0.25,"error":"dial tcp <IP_ADDRESS>:14267: connect: connection refused","stacktrace":"github.com/jaegertracing/jaeger/pkg/discovery/peerlistmgr.(*PeerListManager).ensureConnections\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/pkg/discovery/peerlistmgr/peer_list_mgr.go:171\ngithub.com/jaegertracing/jaeger/pkg/discovery/peerlistmgr.(*PeerListManager).maintainConnections\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/pkg/discovery/peerlistmgr/peer_list_mgr.go:101"}
ubuntu@ip-10-0-2-230:~$ netstat -an | grep 14267
tcp6       0      0 :::14267                :::*                    LISTEN
Could this be issue with tcp and tcp6?

--collector.host-port=<IP_ADDRESS>:14267

On Docker, <IP_ADDRESS> is the container itself, not the host where it's running.

aah. I missed this...thanks

@jpkrohling,
Thanks for your support and time.
Since our Sampling count was 0.001 for one 1000 hit we were getting one trace in Jaeger UI
Initially we have failed to recognize that.
Apart from that everything is working good and application is sending traces.
Thanks :-)

@roopagowda I have the same problem. I can see all trace in local Jaeger UI, but when I put them into different instance, I cant get any trace and my sampler type is const sampler param is 1. Do you have any idea about this problem?

@lxkaka we need more information about your environment. Are you running the all-in-one, or each component on its own host? Are you running on bare metal, or Docker? Where's the agent, in relation to the traced application and in relation to the collector?

@jpkrohling
This the thing. The jager agent has own machine, collector and ui in the another machine. First I try to connect to es backend , but I cant catch any trace. So I try to use collector with memory backend. In the same situation I cant catch any trace.
Here is my docker compose file:
`version: "3"
services:
jaeger-collector:
image: jaegertracing/jaeger-collector
ports:
- "14269:14269"
- "14268:14268"
- "14267:14267"
- "9411:9411"
networks:
- default
restart: on-failure
environment:
- SPAN_STORAGE_TYPE=memory
command: [
#"--es.server-urls=http://<IP_ADDRESS>:9200",
"--memory.max-traces=1000",
"--log-level=debug"
]
#depends_on:
#  - elasticsearch
#jaeger-agent:
image: jaegertracing/jaeger-agent
hostname: jaeger-agent
command: ["--collector.host-port=jaeger-collector:14267"]
ports:
- "5775:5775/udp"
- "6831:6831/udp"
- "6832:6832/udp"
- "5778:5778"
networks:
- elastic-jaeger
networks:
- elastic-jaeger
restart: on-failure
environment:
- SPAN_STORAGE_TYPE=elasticsearch
depends_on:
- jaeger-collector
jaeger-query:
image: jaegertracing/jaeger-query
environment:
- SPAN_STORAGE_TYPE=memory
- no_proxy=localhost
ports:
- "16686:16686"
- "16687:16687"
networks:
- default
restart: on-failure
command: [
#"--es.server-urls=http://<IP_ADDRESS>:9200",
#"--span-storage.type=elasticsearch",
"--memory.max-traces=1000",
"--log-level=debug"
#"--query.static-files=/go/jaeger-ui/"
]
depends_on:
- jaeger-collector log show:jaeger-collector_1  | 2018-08-09T08:52:31.737071735Z {"level":"info","ts":1533804751.736927,"caller":"healthcheck/handler.go:99","msg":"Health Check server started","http-port":14269,"status":"unavailable"}
jaeger-collector_1  | 2018-08-09T08:52:31.737223952Z {"level":"info","ts":1533804751.7371488,"caller":"memory/factory.go:55","msg":"Memory storage configuration","configuration":{"MaxTraces":1000}}
jaeger-collector_1  | 2018-08-09T08:52:31.737855280Z {"level":"info","ts":1533804751.737793,"caller":"static/strategy_store.go:77","msg":"No sampling strategies provided, using defaults"}
jaeger-collector_1  | 2018-08-09T08:52:31.737918697Z {"level":"info","ts":1533804751.7378654,"caller":"collector/main.go:141","msg":"Registering metrics handler with HTTP server","route":"/metrics"}
jaeger-collector_1  | 2018-08-09T08:52:31.737952308Z {"level":"info","ts":1533804751.7379005,"caller":"collector/main.go:149","msg":"Starting Jaeger Collector HTTP server","http-port":14268}
jaeger-collector_1  | 2018-08-09T08:52:31.737958490Z {"level":"info","ts":1533804751.737913,"caller":"healthcheck/handler.go:133","msg":"Health Check state change","status":"ready"}
jaeger-query_1      | 2018-08-09T08:52:32.321901870Z {"level":"info","ts":1533804752.321113,"caller":"healthcheck/handler.go:99","msg":"Health Check server started","http-port":16687,"status":"unavailable"}
jaeger-query_1      | 2018-08-09T08:52:32.323285311Z {"level":"info","ts":1533804752.3231862,"caller":"memory/factory.go:55","msg":"Memory storage configuration","configuration":{"MaxTraces":1000}}
jaeger-query_1      | 2018-08-09T08:52:32.323299988Z {"level":"info","ts":1533804752.3232355,"caller":"query/main.go:180","msg":"Archive storage not created","reason":"Archive storage not supported"}
jaeger-query_1      | 2018-08-09T08:52:32.551971761Z {"level":"info","ts":1533804752.5518212,"caller":"query/main.go:125","msg":"Registering metrics handler with HTTP server","route":"/metrics"}
jaeger-query_1      | 2018-08-09T08:52:32.552059577Z {"level":"info","ts":1533804752.5519016,"caller":"healthcheck/handler.go:133","msg":"Health Check state change","status":"ready"}
jaeger-query_1      | 2018-08-09T08:52:32.552112344Z {"level":"info","ts":1533804752.5520308,"caller":"query/main.go:134","msg":"Starting jaeger-query HTTP server","port":16686}The agent log show:jaeger-agent_1  | 2018-08-09T09:16:57.134805643Z {"level":"info","ts":1533806217.134292,"caller":"tchannel/builder.go:94","msg":"Enabling service discovery","service":"jaeger-collector"}
jaeger-agent_1  | 2018-08-09T09:16:57.134845921Z {"level":"info","ts":1533806217.1343672,"caller":"peerlistmgr/peer_list_mgr.go:111","msg":"Registering active peer","peer":"<IP_ADDRESS>:14267"}
jaeger-agent_1  | 2018-08-09T09:16:57.135284852Z {"level":"info","ts":1533806217.1351295,"caller":"agent/main.go:62","msg":"Starting agent"}
jaeger-agent_1  | 2018-08-09T09:16:58.135179196Z {"level":"info","ts":1533806218.1348429,"caller":"peerlistmgr/peer_list_mgr.go:157","msg":"Not enough connected peers","connected":0,"required":1}
jaeger-agent_1  | 2018-08-09T09:16:58.135204972Z {"level":"info","ts":1533806218.1349018,"caller":"peerlistmgr/peer_list_mgr.go:166","msg":"Trying to connect to peer","host:port":"<IP_ADDRESS>:14267"}
jaeger-agent_1  | 2018-08-09T09:16:58.139015123Z {"level":"info","ts":1533806218.138778,"caller":"peerlistmgr/peer_list_mgr.go:176","msg":"Connected to peer","host:port":"[::]:14267"}`
thanks in advance !

Hi
I think I am having the same issue, i could see that my instrumented applications is reporting the span like these: 2018/08/18 02:50:48 Reporting span 45fddd66abb80684:45fddd66abb80684:0:1, though I could not see the traces on the jaeger ui.
Here is my current setting of the jeager config:
SamplerConfig{Type: "const", Param: 1}, ReporterConfig{LogSpans: true}
Here is how I run my application: docker run --net="mynet" --env-file=./.env --name myapp -p <IP_ADDRESS>:3009:3009 || docker start -i myapp. Then i would start my app's go binary inside the docker interactive shell. After it runs it logs this 2018/08/18 02:50:33 Initializing logging reporter
Here is how I run the jaeger all in one: docker run --name jaeger --net="mynet" -d -p6831:6831/udp -p16686:16686 -p14268:14268 -p14267:14627 jaegertracing/all-in-one:latest
Here is the logs from jaeger all in one:

{"level":"info","ts":1534559978.5076506,"caller":"healthcheck/handler.go:99","msg":"Health Check server started","http-port":14269,"status":"unavailable"}
{"level":"info","ts":1534559978.508828,"caller":"memory/factory.go:55","msg":"Memory storage configuration","configuration":{"MaxTraces":0}}
{"level":"info","ts":1534559978.518636,"caller":"tchannel/builder.go:94","msg":"Enabling service discovery","service":"jaeger-collector"}
{"level":"info","ts":1534559978.5191987,"caller":"peerlistmgr/peer_list_mgr.go:111","msg":"Registering active peer","peer":"<IP_ADDRESS>:14267"}
{"level":"info","ts":1534559978.5268018,"caller":"standalone/main.go:187","msg":"Starting agent"}
{"level":"info","ts":1534559978.5324962,"caller":"standalone/main.go:227","msg":"Starting jaeger-collector TChannel server","port":14267}
{"level":"info","ts":1534559978.5353398,"caller":"standalone/main.go:237","msg":"Starting jaeger-collector HTTP server","http-port":14268}
{"level":"info","ts":1534559979.5207489,"caller":"peerlistmgr/peer_list_mgr.go:157","msg":"Not enough connected peers","connected":0,"required":1}
{"level":"info","ts":1534559979.5392709,"caller":"peerlistmgr/peer_list_mgr.go:166","msg":"Trying to connect to peer","host:port":"<IP_ADDRESS>:14267"}
{"level":"info","ts":1534559979.5800037,"caller":"peerlistmgr/peer_list_mgr.go:176","msg":"Connected to peer","host:port":"[::]:14267"}
{"level":"info","ts":1534559979.9633992,"caller":"standalone/main.go:298","msg":"Registering metrics handler with jaeger-query HTTP server","route":"/metrics"}
{"level":"info","ts":1534559979.976776,"caller":"standalone/main.go:304","msg":"Starting jaeger-query HTTP server","port":16686}
{"level":"info","ts":1534559979.9794478,"caller":"healthcheck/handler.go:133","msg":"Health Check state change","status":"ready"}

I cannot seem to understand which one is failing though, can some one give me some pointers? Thank you!

@adamfadhil have you tried without --net="mynet"? Another option you can try is configure Jaeger client to use HTTP sender instead of UDP, since HTTP sender will be able to log transmission errors.

Hey thanks for the help @yurishkuro . So i've tried it with the LocalAgentHostPort with the previous setting.
Current config: SamplerConfig{Type: "const", Param: 1}, ReporterConfig{LogSpans: true, LocalAgentHostPort: "jaeger:14268"}
Running all in one with docker run -d --name jaeger \   -e COLLECTOR_ZIPKIN_HTTP_PORT=9411 \   -p 5775:5775/udp \   -p 6831:6831/udp \   -p 6832:6832/udp \   -p 5778:5778 \   -p 16686:16686 \   -p 14268:14268 \   -p 9411:9411 \   jaegertracing/all-in-one:latest
I know it worked because if I try to use foobar:14268 on LocalAgentHostPort it would say error no such host.
The log from myapp said that it is still reporting spans, and the logs from jaeger's all in one is still the same from before and no errors. I don't understand how I would run jaeger all in one without the --net="mynet" though, because myapp` is running on it's own docker network.

you don't need the --net setting, but you do need to pass --link jaeger to the application so that  jaeger:<port> can be resolved.
https://github.com/jaegertracing/jaeger/tree/master/examples/hotrod

Thanks again for the answer.
This is how I run myapp currently: docker run --link jaeger --env-file=./.env --name myapp -p <IP_ADDRESS>:3009:3009 || docker start -i myapp. How i ran the jaeger all in one is still the same.
The jaeger UI still does not yet list my service on the dropdown list, though the instrumented app is running as expected like before (reporting spans).
I have not updated my config, so it's the same as before. What I do not understand is that it would still not produce error when I use a different port like before. I'll just put how I initialize my Jaeger below:
func initJaeger(service string) (opentracing.Tracer, io.Closer) {
	cfg := &jaegerconfig.Configuration{
		Disabled: false,
		Sampler: &jaegerconfig.SamplerConfig{
			Type:  "const",
			Param: 1,
		},
		Reporter: &jaegerconfig.ReporterConfig{
			LogSpans:           true,
			LocalAgentHostPort: "jaeger:14268",
		},
	}

	tracer, closer, err := cfg.New(service, jaegerconfig.Logger(jaeger.StdLogger))
	if err != nil {
		panic(fmt.Sprintf("ERROR: cannot init Jaeger: %v\n", err))
	}
	return tracer, closer
}

@yurishkuro Thank you for the answer, I have solved it now.
How I initiate my jaeger jaeger function:
func initJaeger(service string) (opentracing.Tracer, io.Closer) {
	cfg := &jaegerconfig.Configuration{
		Disabled: false,
		Sampler: &jaegerconfig.SamplerConfig{
			Type:  "const",
			Param: 1,
		},
		Reporter: &jaegerconfig.ReporterConfig{
			LogSpans: true,
		},
	}

	var sender jaeger.Transport
	sender = jaegertransport.NewHTTPTransport(
		"http://jaeger:14268/api/traces",
		jaegertransport.HTTPBatchSize(1),
	)

	tracer, closer, err := cfg.New(
		service,
		jaegerconfig.Reporter(
			jaeger.NewRemoteReporter(
				sender,
				jaeger.ReporterOptions.BufferFlushInterval(1*time.Second),
				jaeger.ReporterOptions.Logger(jaeger.StdLogger),
			),
		),
		jaegerconfig.Logger(jaeger.StdLogger),
	)
	if err != nil {
		panic(fmt.Sprintf("ERROR: cannot init Jaeger: %v\n", err))
	}
	return tracer, closer
}

How I run myapp: docker run --link jaeger --env-file=./.env --name myapp -p <IP_ADDRESS>:3009:3009 || docker start -i myapp
How I run jaeger all in one: just follow here

"level":"info","ts":1613572937.3705058,"caller":"peerlistmgr/peer_list_mgr.go:176","msg":"Connected to peer","host:port":"[::]:14267"} ... getting this error on running ./scripts/jaeger-all-in-one.sh command.

I have a similar error
I have three files yml with compose and the reference to docker images are the next:
elasticsearch-docker-compose.yml
version: '3.6'
services:
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:7.10.2
ports:
- '9200:9200'
- '9300:9300'
environment:
- discovery.type=single-node
networks:
- metrics-network
networks:
metrics-network: null
jaegerquery.yaml
version: "3.6"
services:
tracing-query:
image: jaegertracing/jaeger-query:1.24
ports:
- "16686:16686"
environment:
- "TZ=America/Bogota"
- "SPAN_STORAGE_TYPE=elasticsearch"
- "ES_SERVER_URLS=http://elasticsearch:9200"
networks:
- metrics-network
logging:
driver: "json-file"
options:
max-size: "50m"
networks:
metrics-network:
opentelemetry-collector.yml
version: '3.6'
services:
otel-agent:
image: 'otel/opentelemetry-collector:0.47.0'
command:
- '--config=/etc/otel-agent-config.yaml'
- '$${OTELCOL_ARGS}'
volumes:
- './otel-agent-config.yml:/etc/otel-agent-config.yaml'
ports:
- '8887:8888'
- '14250:14250'
- '14268'
- '55678'
- '4317:4317'
- '9411'
- '1777:1777'
- '55679:55679'
- '13133'
networks:
- metrics-network
networks:
metrics-network: null
otel-agent-config.yml
receivers:
otlp:
protocols:
grpc:
opencensus:
jaeger:
protocols:
grpc:
thrift_http:
exporters:
jaeger:
endpoint: tracing-collector:14250
tls:
insecure: true
logging:
loglevel: debug
processors:
batch:
extensions:
pprof:
endpoint: :1777
zpages:
endpoint: :55679
health_check:
service:
extensions: [health_check, pprof, zpages]
pipelines:
traces:
receivers: [otlp, opencensus, jaeger]
processors: [batch]
exporters: [jaeger, logging]
metrics:
receivers: [otlp, opencensus]
processors: [batch]
exporters: [logging]
All the docker images are compatibles with ARM64
but when run the collector with the agent, is generate the next log error:
2022-08-10T02:04:41.735Z        info    service/collector.go:141        Everything is ready. Begin running and processing data.
2022-08-10T02:04:41.942Z        warn    zapgrpc/zapgrpc.go:191  [core] grpc: addrConn.createTransport failed to connect to {tracing-collector:14250 tracing-collector:14250   0 }. Err: connection error: desc = "transport: Error while dialing dial tcp: lookup tracing-collector on <IP_ADDRESS>:53: no such host" {"grpc_log": true}
And I don't what can be.


Challenge 1 - Paul Shell

Added solution for challenge 1.
API is live at: https://cloudexperiment-serverlesschallenge1.azurewebsites.net/dreidel
Blazor client is live at: https://serverlesschallenges.azureedge.net/dreidel

Hi @paul-shell!
Once you have the challenge result, you will compare it to the challenge you created. And after make all the necessary changes in your code, we will close this PR.
Remember that all challenges must be developed in your repository. That's why is very important to fork this repository.
And thank you for your awesome contribution @paul-shell!

Congratulations! This solution has been awarded a Day 1 Winner by your friendly Azure Advocates! Here is your badge! Please reach out to me on Twitter @jenlooper and I will send you a little prize!

We checked out your repository. We're thrilled to say your solution is now listed in our Hall of Fame
Thank you for your contributions!
This PR is now closed! 🎉


Add classification script

Description
(Brief description on what this PR is about)
Checklist
Essentials

[ ] PR's title starts with a category (e.g. [BUGFIX], [MODEL], [TUTORIAL], [FEATURE], [DOC], etc)
[ ] Changes are complete (i.e. I finished coding on this PR)
[ ] All changes have test coverage
[ ] Code is well-documented

Changes

[ ] Feature1, tests, (and when applicable, API doc)
[ ] Feature2, tests, (and when applicable, API doc)

Comments

If this change is a backward incompatible change, why must this change be made.
Interesting edge cases to note here

cc @dmlc/gluon-nlp-team

Would you support tensorboard?

Could you fix this warning in the documentation build so that the CI will pass

[2021-03-05T00:43:22.211Z] /workspace/gluon-nlp/docs/model_zoo/classification/index.rst: WARNING: document isn't included in any toctree

Essentially you need to link the file in https://github.com/dmlc/gluon-nlp/blob/master/scripts/index.rst

The issue [2021-03-10T11:39:50.280Z] /workspace/gluon-nlp/docs/model_zoo/classification/index.rst: WARNING: document isn't included in any toctree still remains

@barry-jin do you think the issue is related to the CI website-build setup?

@barry-jin do you think the issue is related to the CI website-build setup?
Try to add classification/index in toctree

.. toctree::
   :hidden:
   :maxdepth: 1

   benchmarks/index
   classification/index
   conversion_toolkits/index
   datasets/index
   generation/index
   machine_translation/index
   pretraining/index
   processing/index
   question_answering/index


Component assumes that if it's running in Docker it is using Alpine

When I try and work on this component locally in Docker, I'm running into the problem of the _lazy_install() function assuming that all installations in Docker are running on Alpine Linux.
Running HA inside an Ubuntu container and attempting to install the component gives the following result:
Config flow could not be loaded: 500 Internal Server Error Traceback (most recent call last): File "/home/vscode/.local/lib/python3.10/site-packages/aiohttp/web_protocol.py", line 433, in _handle_request resp = await request_handler(request) File "/home/vscode/.local/lib/python3.10/site-packages/aiohttp/web_app.py", line 504, in _handle resp = await handler(request) File "/home/vscode/.local/lib/python3.10/site-packages/aiohttp/web_middlewares.py", line 117, in impl return await handler(request) File "/home/vscode/.local/lib/python3.10/site-packages/homeassistant/components/http/security_filter.py", line 67, in security_filter_middleware return await handler(request) File "/home/vscode/.local/lib/python3.10/site-packages/homeassistant/components/http/forwarded.py", line 100, in forwarded_middleware return await handler(request) File "/home/vscode/.local/lib/python3.10/site-packages/homeassistant/components/http/request_context.py", line 28, in request_context_middleware return await handler(request) File "/home/vscode/.local/lib/python3.10/site-packages/homeassistant/components/http/ban.py", line 80, in ban_middleware return await handler(request) File "/home/vscode/.local/lib/python3.10/site-packages/homeassistant/components/http/auth.py", line 235, in auth_middleware return await handler(request) File "/home/vscode/.local/lib/python3.10/site-packages/homeassistant/components/http/view.py", line 146, in handle result = await result File "/home/vscode/.local/lib/python3.10/site-packages/homeassistant/components/config/config_entries.py", line 148, in post return await super().post(request) File "/home/vscode/.local/lib/python3.10/site-packages/homeassistant/components/http/data_validator.py", line 72, in wrapper result = await method(view, request, data, *args, **kwargs) File "/home/vscode/.local/lib/python3.10/site-packages/homeassistant/helpers/data_entry_flow.py", line 71, in post result = await self._flow_mgr.async_init( File "/home/vscode/.local/lib/python3.10/site-packages/homeassistant/config_entries.py", line 831, in async_init flow, result = await task File "/home/vscode/.local/lib/python3.10/site-packages/homeassistant/config_entries.py", line 849, in _async_init flow = await self.async_create_flow(handler, context=context, data=data) File "/home/vscode/.local/lib/python3.10/site-packages/homeassistant/config_entries.py", line 973, in async_create_flow flow = handler() File "/workspaces/ha_hatch/custom_components/ha_hatch/config_flow.py", line 68, in __init__ _lazy_install() File "/workspaces/ha_hatch/custom_components/ha_hatch/__init__.py", line 74, in _lazy_install _install_alpine_dependencies() File "/workspaces/ha_hatch/custom_components/ha_hatch/__init__.py", line 58, in _install_alpine_dependencies with Popen( File "/usr/local/lib/python3.10/subprocess.py", line 971, in __init__ self._execute_child(args, executable, preexec_fn, close_fds, File "/usr/local/lib/python3.10/subprocess.py", line 1847, in _execute_child raise child_exception_type(errno_num, err_msg, err_filename) FileNotFoundError: [Errno 2] No such file or directory: 'apk'

What it seems to boil down to is that it's unable to find the "apk" command and fully errors out as a result.
I will probably create a PR shortly once I find the right fix, but just wanted to post about the issue in the meantime.

the way I do it is doing intelliJ with simple docker-compose.yml
version: '3'

services:
  homeassistant:
    container_name: homeassistant
    image: "ghcr.io/home-assistant/home-assistant:stable"
    volumes:
      - /Users/dahlb/Desktop/ha_config:/config
      - /etc/localtime:/etc/localtime:ro
    restart: unless-stopped
    ports:
      - 8123:8123

it uses the official docker image just fine.

I think that seems good in general, but with how the component assumes that any Docker container must be running on Alpine, it means nobody running HA in a different type of Docker container would be able to configure the component.
If the component needs to keep the code that runs the apk command, would it make sense to depend on the "distro" library and check if it's running in alpine first?

I think this should be good to go now that Debian and Ubuntu support are added. Thank you!


Why NAND sizes info was deleted?

I've noticed that the information about the NAND Sizes has been removed. Was that intentional? Maybe it was inaccurate and/or outdated?

Decrypt9 has (for a while) verified the size automatically. It's no longer needed to check it manually.


Clustering and point_count

First, let me say how ridiculously great this lib is. I'm having my mind blown every time I use it. We've having lots of success...and that's a credit to your implementation and extensive docs. Really well done.
We have one scenario that has us scratching our heads though and I can't seem to figure out how to make what we're after work (and it's likely a case of being dumb, not understanding the correct options to pass, being dumb again or being extra dumb).
We have a json file that has ~200,000 point features in it (some proprietary data for Wyoming households). We pass that to tippecanoe using -zg --cluster-densest-as-needed --cluster-distance=10 -b0. Then we take the resultant tileset, render it on a map, then use querySourceFeatures to sum the generated point_count properties (realizing that due to the nature of vector tiles, it's very unlikely I'll ever get a 100% accurate count of the source features...in our use-case an estimate is fine).
But in all scenarios, we're ending up with a total point_count of far less than ~200,000.
For example, at zoom level 5, querySourceFeatures returns 33 features with a point_count sum of 38 . Only 3 of those features have clustered=true and they each carry a point_count <= 3 -- the fact that there are only 33 features is generally OK...but we would expect the point_count's to be much higher than what it's generating. As we zoom in, we get more features, but never getting to a total point_count that seems to make sense for having ~200,000 features in the source file.
I've tried a number of different option combos and haven't been able to figure out where I'm going wrong:

-zg --cluster-densest-as-needed --cluster-distance=10 -b0
-zg --cluster-densest-as-needed --cluster-distance=10 -b0 --no-feature-limit
-zg --cluster-distance=10 -b0 --no-feature-limit
-zg --cluster-distance=1 -b0 --no-feature-limit
-zg --cluster-distance=0 -b0 --no-feature-limit
etc. changing the cluster distance value around...we've tried dozens of different options/combos

It very much feels like I'm just missing a critical option to make this work.
Can anyone spot what we're doing wrong?
Thank you so much in advance for any suggestions you may have!

I think the thing you're missing is -r1 to turn off the normal dot-dropping at low zoom levels. I probably should have made the clustering options automatically do that, but it's not automatic now.

Wow...that immediately made an enormous difference! Thanks so much for the quick response.
Now I'm getting counts that are 2 to 3 times what they should be (in fact, with the exception of z0 which is always correct, every zoom level is almost exactly 2 or 3 times larger...meaning if you divide the number of the summed point_count by 2 or 3...you get a number very close to the total number of features...say +/-100).
I initially thought this had to do with duplication of features near tile boundaries, so I generated an id field and de-duplicated...but that didn't help either.
After decoding a zoom level, parsing out the point_count properties and adding them up manually...it does appear that tippecanoe is aggregating correctly. So, I'm guessing I'm up against a problem with querySourceFeatures.
Thanks again for the quick response!
Mike

Great, glad to hear it's working for you now!


No such dash external downloader "native"

Checklist

[ ] I'm reporting a bug unrelated to a specific site
[X] I've verified that I'm running yt-dlp version 2<IP_ADDRESS> (update instructions) or later (specify commit)
[X] I've checked that all provided URLs are playable in a browser with the same IP and same login details
[X] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[X] I've searched the bugtracker for similar issues including closed ones. DO NOT post duplicates
[X] I've read the guidelines for opening an issue

Description
This works with previous release, but with 2<IP_ADDRESS> I got this error.
Verbose log
$ yt-dlp -vU --no-config --downloader "dash,m3u8:native" 'https://www.youtube.com/watch?v=TZbJMUuzmls'
[debug] Command-line config: ['-vU', '--no-config', '--downloader', 'dash,m3u8:native', 'https://www.youtube.com/watch?v=TZbJMUuzmls']

Usage: yt-dlp [OPTIONS] URL [URL...]

yt-dlp: error: No such dash external downloader "native"

Duplicate of https://github.com/yt-dlp/yt-dlp/issues/4157


Configuration code ?

Hello again,
When trying to add my PS4 in HomeKit through add accessory, the PS4 is detected alright but I'm asked for the HomeKit configuration code (8-digits), what should I used?
It didn't work using the pin code provided by PS4 system in add a peripheral for remote play...

I noticed the configuration code actually appear in the homebridge log at the start on PlayStation plugin, along with the PlayStation name.


tensorflow multigpu with dataset api is not converging

I am trying to train a resNet50  on multi_gpus,
I have used
https://github.com/tensorflow/models/blob/master/tutorials/image/cifar10/cifar10_multi_gpu_train.py)
and I have added tensorflow Dataset input pipeline to the link code
But after some steps loss is some value and does not changes any more!(value =0.693147 always)
I have tested this network with dataset API on single gpu,and worked fine.
Im sure something is wrong with my data feeding,because when I feed a simple random nd_array,it converges!!!!!
I have tested my tfrecord file,it was OK .
but I dont know the problem.
I use
ubuntu 16.04,
tensorflow 1.4,
python 2.7.12,
gtx 1080 Gpus
this is all my train code:
github.txt

Quick hint. You need to set reuse to false for gpu:0 and to true for all others

@maxfiedler
I didnt understand you,can you explain with code ?

@maxfiedler
thanks-I did it,but didnt solve the problem

This question is better asked on  StackOverflow since it is not a  bug or feature request. There is also a larger community that reads questions there. Thanks!


Update Applio.ipynb

addition features in the extra selection

what features?

youtube audio downloader (acapella)


Add TraceFlags class

This implement TraceFlags. This used to be called TraceOptions, but https://github.com/open-telemetry/opentelemetry-specification/pull/234 proposes a rename to match the W3C spec.
This also implements generate_span_id and generate_trace_id in SpanContext.

Does it make sense to represent trace and span ids as lowercase hex internally?

It is certainly a reasonable choice. We cheated a little in Shopify's internal tracing project, and used lowercase hex trace IDs, but integers in the range 1..(2**62 - 1) for span IDs. The latter are generated more often and can be stored more efficiently (small integer).
We can use Random::DEFAULT.bytes(16).unpack1('H*') to generate trace IDs, however we have to wrap this in a loop, checking for all 0.

I think going with a string for our internal representation is a good approach for now. If it turns we regularly need both string and integer representations, we can consider #23.

Bumping this comment, in case it was missed.

I considered this, but couldn't find a convenient and obvious place for them. We don't currently have any methods on the OpenTelemetry::Trace module, for example, and the OpenTelemetry::Trace::Tracer class is meant to be a no-op implementation. Should we make it part of the Tracer API, and allow alternative implementations to provide their own?

Added tests for SpanContext to exercise these codepaths, and fixed the bugs they uncovered.

This is ready for review.

can this be approved now?  It looks as though the comments have been addressed?


3DJake PLA Spool // also no clue how to pull ;)

3DJake
ecoPLA White
1,75mm
1000g
Spoolweight: 234g (measured 3 times)

btw thanks for you effort, highly appreciated

if you make a screencast/screenrecorder how to make a pull-request, would ne nice... so next time i'll safe you some time ;)

i posted it on subreddit "3dprinting" https://www.reddit.com/r/3Dprinting/comments/13cjpds/spool_database_with_weights_and_size_via_git/
more people need this and also contribute ;)

This is great way how to do it :)

I will try to prepare something. In the mean time, this is almost as fast
Thank you for your contrib -- it's published now.


Automate deploy (and, eventually, provision)

The Digital Ocean droplet was set manually and the only piece of software that makes deploy easier is a git hook. Yet, nginx and gunicorn processes are owned by root — that's not cool.
Something like Ansible might help.
Personally I don't have devops know-how to fix the users and processes issues, and I've never used Ansible — so any help is appreciated. Surely I can share the application specific logic and help anyone willing to close that issue ; )

Velho, posso auxiliar você a montar deploy de ambientes complexos usando Docker.
É possível manipular nginx e outros processos que precisam de root, utilizando containers do Docker.
Sem precisar ter super acesso da VPS na DigitalOcean.

Great! I have little experience with Docker but with a tutor I guess I can use it properly. When are you available to talk about it?

Boa! Eu tenho pouca experiência com Docker, mas com um tutor eu acho que eu consigo usar certinho. Quando tu pode conversar sobre isso?

Me passa o seu telefone ou liga pra mim. (48) 8810-8161
A ideia é Dockerrizar o seu projeto e automatizar o deploy via containers, fazendo o build e criando um container com a versão final de produção do seu projeto, seja ele fullstrack ou SPA.
Possível até versionar na nuvem esses containers, facilitando assim rollbacks nos deploys quando necessário.
Abraços.

Sure thing — I'm gonna drop you a line at your Gmail.
I've played around with Docker, so I get the idea. I just don't know how to move from the theory to practice.
Anyway, thanks for the help, gonna follow up via email.

How this one goes? The idea is to create a dockerfile and use travis to build image and deploy the application?

@vitalan, I think yes. The ideia is delivery the app in a container, we can any CI service to build a image, save on registry and after deploy the app.

@cuducos can I help you too? My Gmail is<EMAIL_ADDRESS>

Many thanks, @vitallan and @gwmoura for jumping in!
This Tuesday I was in a call with @luiz-simples and he's probably working on a branch to fix this issue.
In baby steps probably we'll start just with Docker for a dev environment to later integrate with CI (Travis is already in use for tests here, so probably we'll stick to Travis) and automate the deploy.
Feel free to talk to @luiz-simples and you guys can find the best way to help him.
I'm gonna to introduce you all via email. And count on me for doubts on the repo/app, on pairing etc.

@vitallan @gwmoura @cuducos May I help? (:

I could help with the provision scripts of the
Droplet, creating the ansible scripts to deploy
the docker containers Luiz want to create.

Looks good to me. I'd love to learn bits of Ansible — it's in my TODO list anyway…

How this architecture must be? Something like nginx, python-cgi and some database?

@luiz-simples you'll need something to orchestrate the containers, main it hard coded is a waste of time.
In my opinion, something like Rancher (hosted) or AWS EC2 (PaaS) is alot better (I can help in both).

@pedrommone many thanks!

How this architecture must be?

I have almost no experience with Docker, but I'll try to help, please follow up if what I'm saying is non-sense. This install steps (i.e. everything we need to do to get stuff running) is detailed in the README.md, so what I imagine is:

A container to run Python 3 (but I have no idea what python-cgi is about), this will run basically Django and gunicorn
A container to run NodeJS and generate a set of static files (the front-end is in Elm, agnostic about backend and we should even split it into a separate repository, not sure about how to handle this at this pointm but check my comment on #18)
A container able to run nginx to route the app requests to Django container and static files requests to the file system (static files are generated by the Python/Django container and by the NodeJS container, but they don't need to be at the same path within the file system)
A container to the database (Postgres)

Is this of any help?

In my opinion, something like Rancher (hosted) or AWS EC2 (PaaS) is alot better (I can help in both).

We used to use a PaaS but now we are powered (supported, sponsored) by Digital Ocean, so we are deploying to a Digital Ocean droplet, is that ok?

@cuducos thanks for the awesome info.
About the DO, is totally fine. I can help you with that and make something working well. As soon I get some time work on the containers.
I will trace the following steps

[ ] Build the docker images
[ ] Build the docker-compose.yml
[ ] Build the DO infrastructure with Rancher (personal choice, we use it here.)
[ ] Build the deploy pipeline and make staging and production environment too
[ ] Make sure everything is working well and migrate all the stuff

What do you guys think?

Two minor doubts:

As a newbie to Docker world I haven't heard of Rancher before — what problem does it target? I watched the welcome video but probably I miss the limitations of the environment to understand the solution Rancher offers (I mean, in my naïve point of view, docker-compose.yml would wire up the infrastructure; so maybe the idea is to simplify deploying to different environments, but then… again… I have no idea how it's done without Rancher because I'm a newbie hahaha).
What do you all think is better: to split backend and front-end before we work on the Docker stuff, to split while we create this new Docker infrastructure, or to get it going with Docker and then split?

@cuducos here we go:

Rancher let you orchestrate your containers, each project is called stack and it let you handle everything via API, it just rocks! You can see more (here)[https://github.com/rancher/rancher].
The promise of rancher is to separate each process into container, one for postgress, one for nginx, one for python.

Here is an exemple of two services

@pedrommone this steps are this same, for me is good option, I don't work with hancher but I listen good opinions :smile:. @luiz-simples is working in create a base image for the project e images for dev and production, when possible looks the branch.
@cuducos

What do you all think is better: to split backend and front-end before we work on the Docker stuff, to split while we create this new Docker infrastructure, or to get it going with Docker and then split?

I think better start creating a Docker env for development to help more people to use and contribute with the project and after we split the project in backend and front-end

@gwmoura you can easily make everything up with docker-compose.yml. In my opinion, make dev and prod environment separately is a waste of time since everything needs to walk together.

I think better too, I prefer create everything on docker-compose.yml, as talk in 12factor.net - prod and dev should be the same. @luiz-simples is using docker-compose.yml and created some scripts to help and automate the deploy.
I am a developer, then I don't know if creating a simple container can impact negative in production... we can work together and create a simple and efficient environment

@pedrommone I know, you don't need write :smile:. Looks the branch  -  https://github.com/luiz-simples/jarbas/tree/docker-environment and comments what you think we need remove

@cuducos witch python and django version we are using?

Python 3.5.2, Django is the latest version (1.10.something specified at requirements-dev.txt)

Now that the Dockerfile was built, the next step is to create the Ansible role (or something similar to ansible) to automatically make the provision of the droplet? Does this correct?

I must say I'm kind of lost here. I guess @pedrommone got stuck at the issue with the CI services — but honestly I could not understand this part because my knowledge on Docker environment is null.
As my idea was to have a working Dockerfile and docker-compose.yml locally I got that having Elm (NodeJS) being called from within Django was an issue, thus I fixed it in a separated branch. I also volunteered to change Travis CI for a different service, but no one could me point to a specific service yet.
@pedrommone can you update the checkboxes and clarify what's holding us? Just asking because maybe more people is willing to jump in and help here — please, don't get me wrong, no pressure, ok?

@cuducos dont worrry :).
I've stuck into dependencies, we cannot have nodejs and python on the same image -- thats not the Docker goal --. When I get some time I'll work on that.

Hey!
I wanna help too!  I know a little bit about docker and ansible. I will study the infra of datasciencebr.
IMHO we don't need to think about rancher or something like that yet.  We can use some service to provide this infra for us: https://hyper.sh/ (for example! I didn't tested yet!).

Hi @gomex — I was about to read you book to try to fix that haha… well I'm gonna read it anyway. The point is that here we do not necessarily need Docker, although I'm pretty sure Docker could help.
I'm mostly a developer, and I'm pretty much familiar with Jarbas — count on me if you have any doubt about the stack (BTW taking about Docker it might be interesting to use the extract-nodejs branch — there compiling assets is done on NodeJS CLI, not within Django pipeline; I'll update that branch now because it was left behind a couple of weeks ago). Unfortunately I'm a newbie in devops and Docker. Therefore I do appreciate your help ; )

@cuducos first thing, is possible pass the username and password for this command?
python manage.py createsuperuser

Can we specify it using environment variable?

Hi @gomex — basically we can skip that. It's optional and we don't have a lot of heavy stuff to do on Django Admin. People who would use it will find their way to do it. Yet there's no way to pass the password (the best we can do is set a super user without password with --noinput — which I wouldn't recommend, too unsafe).

I am trying to use SQLite to avoid Postgres configuration for now and I got this error:
ValueError: SQLite backend does not support timezone-aware datetimes when USE_TZ is False.

I tried to add USE_TZ=True inside .env and didn't change that error :(

I'm sorry about that. Uncaught error probably. Gonna take a look later
today and update you here.
On Wed, 9 Nov 2016 at 12:16 Rafael Gomes<EMAIL_ADDRESS>wrote:

I am trying to use SQLite to avoid Postgres configuration for now and I
got this error:
ValueError: SQLite backend does not support timezone-aware datetimes when USE_TZ is False.
I tried to add USE_TZ=True inside .env and didn't change that error :(
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/datasciencebr/jarbas/issues/12#issuecomment-259424174,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AEg38_7L6jMtmYtef1JU_yAo6aL3gzgFks5q8dWtgaJpZM4KDLVr
.

You can check using my Dockerfile on that repos:
https://github.com/gomex/jarbas
You can reproduce using this command:
docker build -t serenata-jarbas .

Thanks!

Hi @gomex,  bug caught and squashed in 17d1f7f (within extract-nodejs branch)— I tested it here with SQLite and it seems do be working.

Is there any way to build a "small" version of this data? When I try "python manage.py loaddatasets" and "python manage.py loadsuppliers" It takes too long.

Not in in a seamless way.  Once it's there you call these commands with --source path/to/small?verions/directory — does that work?

@cuducos let's try :)

Here it is: download the files and point both scripts to this folder with -s https://www.dropbox.com/sh/cxuizj33rhb4c1n/AAAoT0qk0GGO1Fef-bf2Jj0aa?dl=0

I tried that command:
python manage.py loaddatasets --source ./datasets

I got this error:
Starting with 0 documents
Loading ./datasets/2016-08-08-current-year.xz
Traceback (most recent call last):
  File "manage.py", line 22, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 367, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 359, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/base.py", line 294, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/base.py", line 345, in execute
    output = self.handle(*args, **options)
  File "/code/jarbas/core/management/commands/loaddatasets.py", line 30, in handle
    self.bulk_create_by(documents, options['batch_size'])
  File "/code/jarbas/core/management/commands/loaddatasets.py", line 87, in bulk_create_by
    for document in documents:
  File "/code/jarbas/core/management/commands/loaddatasets.py", line 40, in documents_from
    yield Document(**self.serialize(row))
  File "/usr/local/lib/python3.5/site-packages/django/db/models/base.py", line 555, in __init__
    raise TypeError("'%s' is an invalid keyword argument for this function" % list(kwargs)[0])
TypeError: '' is an invalid keyword argument for this function

I'm sorry about that, @gomex. I couldn't get back to you earlier.
Anyway I was pondering here: these commands are basically a step of provision, of seeding the database. To test if the environment is set properly and if the app is working you can just skip them and run the tests ($ python manage.py test). If they pass and the equivalent of http://localhost gives you a HTTP 200 I guess you're good to go.
Meanwhile I'll try to understand why the smaller datasets haven't worked and update you all soon (I hope so)

@gomex, a few comments after testing your Dockerfile — probably newbie doubts, but anyway:

this branch (extract-nodejs) compiles Elm files (to .js) outside Django, thus instead of $ python manage.py assets build we need a $ npm run assets — and this might need a different container I guess (but all in all, if we go back to master and use $ python manage.py assets build Django will look for NodeJS bin anyway). Is it time to move to Docker Compose already?

Python package psycopg2 will fail (i.e. $ python -m pip install -r requirements-dev.txt); it requires a client version of Postgres installed in the machine (e.g. apt-get postgresql postgresql-contrib); if we're following a baby steps strategy, we can comment it out for a while as you told me we're using SQLite to get started. Not sure about how to fix that issue in a container mindset. Any ideas?

By now I'm just skipping $ python manage.py loaddatasets and $ python manage.py loadsuppliers — once I can reach the Django in a container, I move forward to seeding the database. Does that make sense?

I forgot: we need the --noinput argument to make collectstatic override user input (i.e. $ python manage.py collectstatic --noinput)

@cuducos about your questions:
This branch (extract-nodejs) compiles Elm files (to .js) outside Django, thus instead of $ python manage.py assets build we need a $ npm run assets — and this might need a different container I guess (but all in all, if we go back to master and use $ python manage.py assets build Django will look for NodeJS bin anyway). Is it time to move to Docker Compose already?

I was trying to start from Python, but the ideia is use another image to build this elm assets. The proposal is build inside onde specific folder that should be mount as volume in python container too.
Python container will access that folder and can see elm assets there.
Python container will show me something without elm assets?  if not, what folders should I mount in both container?
Python package psycopg2 will fail (i.e. $ python -m pip install -r requirements-dev.txt); it requires a client version of Postgres installed in the machine (e.g. apt-get postgresql postgresql-contrib); if we're following a baby steps strategy, we can comment it out for a while as you told me we're using SQLite to get started. Not sure about how to fix that issue in a container mindset. Any ideas?

We can install postgresql inside python container for now.
By now I'm just skipping $ python manage.py loaddatasets and $ python manage.py loadsuppliers — once I can reach the Django in a container, I move forward to seeding the database. Does that make sense?

For it is ok! :)

Hey @cuducos
I commented this commands:
#RUN python manage.py loaddatasets --source ./datasets
#RUN python manage.py loadsuppliers
#RUN python manage.py ceapdatasets
#RUN python manage.py assets build
#RUN python manage.py collectstatic
And try to run "python manage.py test" and I got this error:
Creating test database for alias 'default'...
/usr/local/lib/python3.5/site-packages/django/db/models/fields/__init__.py:1430: RuntimeWarning: DateTimeField Document.issue_date received a naive datetime (1970-01-01 00:00:00) while time zone support is active.
  RuntimeWarning)
......./usr/local/lib/python3.5/site-packages/django/db/models/fields/__init__.py:1430: RuntimeWarning: DateTimeField Supplier.last_updated received a naive datetime (2016-11-11 01:17:01.431206) while time zone support is active.
  RuntimeWarning)
.................................FF............
======================================================================
FAIL: test_serializer (jarbas.core.tests.test_loadsuppliers_command.TestSerializer)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/code/jarbas/core/tests/test_loadsuppliers_command.py", line 51, in test_serializer
    self.assertEqual(self.command.serialize(supplier), expected)
AssertionError: {'lon[37 chars]te': datetime.date(1969, 12, 31), 'opening': d[104 chars] 31)} != {'lon[37 chars]te': '1969-12-31', 'opening': '1969-12-31', 'l[59 chars]-31'}
  {'email': None,
   'latitude': 3.1415,
   'longitude': -42.0,
-  'opening': datetime.date(1969, 12, 31),
-  'situation_date': datetime.date(1969, 12, 31),
+  'opening': '1969-12-31',
+  'situation_date': '1969-12-31',
-  'special_situation_date': datetime.date(1969, 12, 31)}
?                            ^^^^^^^^^^^^^^    ^^  ^^  ^

+  'special_situation_date': '1969-12-31'}
?                            ^    ^  ^  ^

======================================================================
FAIL: test_to_date (jarbas.core.tests.test_loadsuppliers_command.TestSerializer)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/code/jarbas/core/tests/test_loadsuppliers_command.py", line 30, in test_to_date
    self.assertEqual(self.command.to_date('22/7/91'), expected)
AssertionError: datetime.date(1991, 7, 22) != '1991-07-22'

----------------------------------------------------------------------
Ran 54 tests in 0.497s

FAILED (failures=2)
Destroying test database for alias 'default'...

Indeed. When we changed USE_TZ to True that started to happen. Fixed in 67069d7. I'm fully available today, @gomex — if you want to pair to speed things up, I'm up to ; )
Addressing your questions (let me know if accidentally I leave any of them behind):

Python container will show me something without elm assets?

Running the app without the NodeJS part will not show anything in the browser except the proper title (i.e. <title>Jarbas | Serenata de Amor</title>) with in the HTML the <head>. If we get to that point, we're doing things right Python wise. Added some tests for that in 1a083ca .

If not, what folders should I mount in both container?

Django collect all static files in a external directory (useful to serve them via nginx, for example). I believe the Elm files could compile to that same dir. But this is just an idea. We could use a different dir and add some lines to nginx conf (i.e. having /static/ for Django collected static files and /elm/ for Elm assets).

We can install postgresql inside python container for now.

I added RUN apt-get install -y postgresql postgresql-contrib but got:
E: Unable to locate package postgresql
E: Unable to locate package postgresql-contrib

Not sure what to do… any clues?

@cuducos we can pair, but I am hospitalized now. I will do a surgery on my collarbone today. Can we try to do that in few days?
Thanks!

About your questions:
I added RUN apt-get install -y postgresql postgresql-contrib but got:

E: Unable to locate package postgresql
E: Unable to locate package postgresql-contrib

You need to update before install:
RUN apt-get update && apt-get install -y postgresql postgresql-contrib

Try that!

OMG I had no clue you were in the hospital. No worries, mate! I'm going through a surgery later this month too. Get well and we talk later ; )
However, let me tell you the good news:
$ docker run jarbas python manage.py test                                        [jarbas]cuducos@thomas
.........................................................
----------------------------------------------------------------------
Ran 57 tests in 1.024s

OK

Yay 🎉
Here it is the Dockerfile I used:
FROM python:3.5
COPY requirements-dev.txt /requirements-dev.txt
RUN python -m pip install -r requirements-dev.txt
COPY ./ /code
WORKDIR /code
RUN apt-get update && apt-get install -y postgresql postgresql-contrib
RUN python manage.py migrate
# RUN python manage.py createsuperuser
# RUN python manage.py loaddatasets
# RUN python manage.py loadsuppliers
# RUN python manage.py ceapdatasets
# RUN python manage.py assets build
# RUN python manage.py collectstatic

@cuducos yeah!!!!
Now we need to configure the gunicorn, right?
How can I open the app and see this title?

Guys testing the Dockerfile pasted here, I received 2 errors, because the elm: FileNotFoundError: [Errno 2] No such file or directory: 'elm-make'
I added more commands on Dockerfile to install elm-make
My Dockerfile
FROM python:3.5
COPY requirements-dev.txt /requirements-dev.txt
RUN python -m pip install -r requirements-dev.txt
COPY ./ /code
WORKDIR /code
RUN apt-get update && apt-get install -y postgresql \
                                         postgresql-contrib \
                                         nodejs \
                                         npm
RUN python manage.py migrate
RUN ln -s /usr/bin/nodejs /usr/bin/node
RUN npm install
# RUN python manage.py createsuperuser
# RUN python manage.py loaddatasets
# RUN python manage.py loadsuppliers
# RUN python manage.py ceapdatasets
# RUN python manage.py assets build

Running the tests:
$ docker run --rm jarbas python manage.py test
....................................................EE..
======================================================================
ERROR: test_status_code (jarbas.frontend.tests.test_home_view.TestGet)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/site-packages/webassets/filter/__init__.py", line 514, in subprocess
    shell=os.name == 'nt')
  File "/usr/local/lib/python3.5/subprocess.py", line 947, in __init__
    restore_signals, start_new_session)
  File "/usr/local/lib/python3.5/subprocess.py", line 1551, in _execute_child
    raise child_exception_type(errno_num, err_msg)
FileNotFoundError: [Errno 2] No such file or directory: 'elm-make'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/code/jarbas/frontend/tests/test_home_view.py", line 8, in test_status_code
    resp = self.client.get(resolve_url('home'))
  File "/usr/local/lib/python3.5/site-packages/django/test/client.py", line 529, in get
    **extra)
  File "/usr/local/lib/python3.5/site-packages/django/test/client.py", line 333, in get
    return self.generic('GET', path, secure=secure, **r)
  File "/usr/local/lib/python3.5/site-packages/django/test/client.py", line 409, in generic
    return self.request(**r)
  File "/usr/local/lib/python3.5/site-packages/django/test/client.py", line 494, in request
    six.reraise(*exc_info)
  File "/usr/local/lib/python3.5/site-packages/django/utils/six.py", line 686, in reraise
    raise value
  File "/usr/local/lib/python3.5/site-packages/django/core/handlers/exception.py", line 39, in inner
    response = get_response(request)
  File "/usr/local/lib/python3.5/site-packages/django/core/handlers/base.py", line 187, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/usr/local/lib/python3.5/site-packages/django/core/handlers/base.py", line 185, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/code/jarbas/frontend/views.py", line 10, in home
    return render(request, 'frontend/home.html', context=context)
  File "/usr/local/lib/python3.5/site-packages/django/shortcuts.py", line 30, in render
    content = loader.render_to_string(template_name, context, request, using=using)
  File "/usr/local/lib/python3.5/site-packages/django/template/loader.py", line 68, in render_to_string
    return template.render(context, request)
  File "/usr/local/lib/python3.5/site-packages/django/template/backends/django.py", line 66, in render
    return self.template.render(context)
  File "/usr/local/lib/python3.5/site-packages/django/template/base.py", line 208, in render
    return self._render(context)
  File "/usr/local/lib/python3.5/site-packages/django/test/utils.py", line 94, in instrumented_test_render
    return self.nodelist.render(context)
  File "/usr/local/lib/python3.5/site-packages/django/template/base.py", line 994, in render
    bit = node.render_annotated(context)
  File "/usr/local/lib/python3.5/site-packages/django/template/base.py", line 961, in render_annotated
    return self.render(context)
  File "/usr/local/lib/python3.5/site-packages/django_assets/templatetags/assets.py", line 87, in render
    for url in bundle.urls():
  File "/usr/local/lib/python3.5/site-packages/webassets/bundle.py", line 787, in urls
    urls.extend(bundle._urls(new_ctx, extra_filters, *args, **kwargs))
  File "/usr/local/lib/python3.5/site-packages/webassets/bundle.py", line 746, in _urls
    *args, **kwargs)
  File "/usr/local/lib/python3.5/site-packages/webassets/bundle.py", line 600, in _build
    force, disable_cache=disable_cache, extra_filters=extra_filters)
  File "/usr/local/lib/python3.5/site-packages/webassets/bundle.py", line 524, in _merge_and_apply
    kwargs=item_data)
  File "/usr/local/lib/python3.5/site-packages/webassets/merge.py", line 276, in apply
    return self._wrap_cache(key, func)
  File "/usr/local/lib/python3.5/site-packages/webassets/merge.py", line 218, in _wrap_cache
    content = func().getvalue()
  File "/usr/local/lib/python3.5/site-packages/webassets/merge.py", line 251, in func
    getattr(filter, type)(data, out, **kwargs_final)
  File "/usr/local/lib/python3.5/site-packages/webassets_elm/__init__.py", line 39, in input
    self.subprocess(write_args, fake_write_obj)
  File "/usr/local/lib/python3.5/site-packages/webassets/filter/__init__.py", line 516, in subprocess
    raise FilterError('Program file not found: %s.' % argv[0])
webassets.exceptions.FilterError: Program file not found: elm-make.

======================================================================
ERROR: test_js (jarbas.frontend.tests.test_static_files.TestCompiledStatic)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/unittest/mock.py", line 1157, in patched
    return func(*args, **keywargs)
  File "/code/jarbas/frontend/tests/test_static_files.py", line 27, in test_js
    call_command('assets', 'build', 'elm', '--no-cache')
  File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 130, in call_command
    return command.execute(*args, **defaults)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/base.py", line 345, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.5/site-packages/django_assets/management/commands/assets.py", line 87, in handle
    'a non-zero exit code (%d).' % retval)
django.core.management.base.CommandError: The webassets build script exited with a non-zero exit code (2).

----------------------------------------------------------------------
Ran 56 tests in 2.027s

FAILED (errors=2)
Creating test database for alias 'default'...
Destroying test database for alias 'default'...

@gomex Good to hear from you! Are you recovering well? I really hope so! Starting the app is done with commands such as gunicorn jarbas.wsgi:application --reload --bind <IP_ADDRESS>:8001 --workers 4 within the repo root directory (and if that's the case, within the project virtualenv). If it's working the app would be available at <IP_ADDRESS>:8001.
@gwmoura The erros are there probably because you're using the master branch (NodeJS parts tangled up within Django), not the recommended extract-nodejs branch (that isolates NodeJS parts from Django). Can you confirm if that's the case?

@cuducos it's the my case, I gonna change my branch and test once more.
Thanks and sorry 😅

Hey @cuducos
I am trying to build the elm assets and static files.
"python manage.py assets build" and "python manage.py collectstatic" are enough or should I do something else? Where are these files?

Hi @gomex,
The commands are a little bit different in extract-nodejs branch.
npm run assets should be run in a container with the NodeJS (the one in which we ran npm install previously). This is supposed to create the file jarbas/frontend/static/app.js (but I'm not sure how to orchestrate something like that with Docker containers).
python manage.py collectstatic should be run in the Django container, and this command will collect all static files from all Django apps and put them in staticfiles/.
One option maybe is to:

change gulpfile.js so it generates the app.js directly into staticfiles/app.js
first run python manage.py collectstatic (to create the staticfiles/ directory)
then run npm run assets and then app.js will be created in the directory in which Django/nginx expects to find it (i.e. staticfiles/)

Does that make sense?

@cuducos @gomex  using a container for NodeJS and running npm run assets a error is showed.
repo: https://github.com/gwmoura/jarbas/tree/extract-nodejs
Dockerfile used for create a nodejs container:
# Dockerfile-node
FROM node:6.9.1
COPY elm-package.json elm-package.json
COPY package.json package.json
RUN npm i
COPY . /code
WORKDIR /code
VOLUME /code
RUN npm install gulp gulp-elm gulp-rename gulp-uglify elm
RUN npm install -g gulp
CMD npm run assets

The error after execute the container with the command npm run assets
$ docker run --rm -v "`pwd`":/code nodejarbas
npm info it worked if it ends with ok
npm info using<EMAIL_ADDRESS>npm info using<EMAIL_ADDRESS>npm info lifecycle @~preassets: @
npm info lifecycle @~assets: @

> @ assets /code
> gulp elm

[13:56:08] Using gulpfile /code/gulpfile.js
[13:56:08] Starting 'elm'...
events.js:160
      throw er; // Unhandled 'error' event
      ^

Error: spawn /code/node_modules/elm/Elm-Platform/0.17.1/.cabal-sandbox/bin/elm-make ENOENT
    at exports._errnoException (util.js:1026:11)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:193:32)
    at onErrorNT (internal/child_process.js:359:16)
    at _combinedTickCallback (internal/process/next_tick.js:74:11)
    at process._tickCallback (internal/process/next_tick.js:98:9)
    at Module.runMain (module.js:606:11)
    at run (bootstrap_node.js:394:7)
    at startup (bootstrap_node.js:149:9)
    at bootstrap_node.js:509:3
Compile error

/node_modules/q/q.js:155
                throw e;
                ^
Error: Compile error

npm info lifecycle @~assets: Failed to exec assets script
npm ERR! Linux 4.4.0-47-generic
npm ERR! argv "/usr/local/bin/node" "/usr/local/bin/npm" "run" "assets"
npm ERR! node v6.9.1
npm ERR! npm  v3.10.8
npm ERR! code ELIFECYCLE
npm ERR! @ assets: `gulp elm`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the @ assets script 'gulp elm'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the  package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     gulp elm
npm ERR! You can get information on how to open an issue for this project with:
npm ERR!     npm bugs
npm ERR! Or if that isn't available, you can get their info via:
npm ERR!     npm owner ls
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR!     /code/npm-debug.log

Hi @gwmoura.
I just tested with a Dockerfile slightly different form yours and it worked:
FROM node:6.9.1
COPY elm-package.json elm-package.json
COPY package.json package.json
RUN npm install -g gulp
RUN npm install
COPY . /code
WORKDIR /code
VOLUME /code
CMD npm run assets

It raised no error and created the expected app.js file:
$ docker run jarbas ls jarbas/frontend/static
app.js
ceap-datasets.html
digitalocean.png

I ran it in the extract-nodejs. If this still failing we can pair to troubleshoot together…

@cuducos your Dockerfile not work for me :(, before build the container you deleted the file app.js?
$ cat Dockerfile-node
FROM node:6.9.1
COPY elm-package.json elm-package.json
COPY package.json package.json
RUN npm install -g gulp
RUN npm install
COPY . /code
WORKDIR /code
VOLUME /code
CMD npm run assets
george@george-ubuntu:~/workspace/datasciencebr/jarbas$ docker build -t nodejarbas -f Dockerfile-node .
Sending build context to Docker daemon 365.6 MB
Step 1 : FROM node:6.9.1
 ---> 130ebee906ca
Step 2 : COPY elm-package.json elm-package.json
 ---> Using cache
 ---> d336bf466a4a
Step 3 : COPY package.json package.json
 ---> Using cache
 ---> ef4015364789
Step 4 : RUN npm install -g gulp
 ---> Using cache
 ---> 4670f4553021
Step 5 : RUN npm install
 ---> Using cache
 ---> ca31fc97fee5
Step 6 : COPY . /code
 ---> Using cache
 ---> 9a61c46c5b97
Step 7 : WORKDIR /code
 ---> Using cache
 ---> dc25b7a5fd37
Step 8 : VOLUME /code
 ---> Using cache
 ---> 126c30a69517
Step 9 : CMD npm run assets
 ---> Using cache
 ---> bbdc5dd4234f
Successfully built bbdc5dd4234f
george@george-ubuntu:~/workspace/datasciencebr/jarbas$ docker run --rm nodejarbas ls jarbas/frontend/static
app.css
digitalocean.png
george@george-ubuntu:~/workspace/datasciencebr/jarbas$ docker run --rm nodejarbas
npm info it worked if it ends with ok
npm info using<EMAIL_ADDRESS>npm info using<EMAIL_ADDRESS>npm info lifecycle @~preassets: @
npm info lifecycle @~assets: @

> @ assets /code
> gulp elm

[14:47:01] Using gulpfile /code/gulpfile.js
[14:47:01] Starting 'elm'...
events.js:160
      throw er; // Unhandled 'error' event
      ^

Error: spawn /code/node_modules/elm/Elm-Platform/0.17.1/.cabal-sandbox/bin/elm-make ENOENT
    at exports._errnoException (util.js:1026:11)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:193:32)
    at onErrorNT (internal/child_process.js:359:16)
    at _combinedTickCallback (internal/process/next_tick.js:74:11)
    at process._tickCallback (internal/process/next_tick.js:98:9)
    at Module.runMain (module.js:606:11)
    at run (bootstrap_node.js:394:7)
    at startup (bootstrap_node.js:149:9)
    at bootstrap_node.js:509:3
Compile error

/node_modules/q/q.js:155
                throw e;
                ^
Error: Compile error

npm info lifecycle @~assets: Failed to exec assets script
npm ERR! Linux 4.4.0-47-generic
npm ERR! argv "/usr/local/bin/node" "/usr/local/bin/npm" "run" "assets"
npm ERR! node v6.9.1
npm ERR! npm  v3.10.8
npm ERR! code ELIFECYCLE
npm ERR! @ assets: `gulp elm`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the @ assets script 'gulp elm'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the  package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     gulp elm
npm ERR! You can get information on how to open an issue for this project with:
npm ERR!     npm bugs
npm ERR! Or if that isn't available, you can get their info via:
npm ERR!     npm owner ls
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR!     /code/npm-debug.log

@cuducos @gomex I needed install elm globally npm install -g elm
npm run assets
npm info it worked if it ends with ok
npm info using<EMAIL_ADDRESS>npm info using<EMAIL_ADDRESS>npm info lifecycle @~preassets: @
npm info lifecycle @~assets: @

> @ assets /code
> gulp elm

[15:17:02] Using gulpfile /code/gulpfile.js
[15:17:02] Starting 'elm'...
Starting downloads...

  ● debois/elm-parts 4.0.0
  ● elm-community/elm-test 2.1.0
  ● debois/elm-dom 1.2.2
  ● debois/elm-mdl 7.6.0
  ● elm-lang/navigation 1.0.0
  ● elm-lang/dom 1.1.0
  ● mgold/elm-random-pcg 3.0.4
  ● NoRedInk/elm-decode-pipeline 2.0.0
  ● elm-community/elm-lazy-list 1.3.0
  ● elm-lang/mouse 1.0.0
  ● elm-lang/core 4.0.5
  ● evancz/elm-http 3.0.1
  ● elm-community/json-extra 1.1.0
  ● elm-lang/html 1.1.0
  ● elm-lang/virtual-dom 1.1.1
  ● elm-community/shrink 1.1.0
  ● elm-lang/window 1.0.0
  ● elm-lang/lazy 1.0.0
  ● rtfeldman/node-test-runner
 2.0.1
Packages configured successfully!
Success! Compiled 98 modules.
Successfully generated /code/1161016-65-1hu8yar.4mxcsug14i.js
[15:18:27] Finished 'elm' after 1.42 min
npm info lifecycle @~postassets: @
npm info ok
root@fea568956089:/code# ls jarbas/frontend/static/
app.css  app.js  digitalocean.png

Yep, you're right: I haven't deleted the app.js… it wasn't working. My bad.

I needed install elm globally npm install -g elm

Good that it fixed, @gwmoura , but it shouldn't be that way. I have to explore the issue deeper I guess. Here are my concerns:

Elm should be installed by npm install (that reads packages.json) to mae sure we're using the right version of Elm (e.g. if we install Elm 0.16 or Elm 0.18 the elm-make will fail).
We could use something like npm install -g<EMAIL_ADDRESS>but we would be repeating ourselves (Elm version would be in Dockerfile and in packages.json)

Maybe a solution:
When we install Elm locally elm-make goes to node_modules/.bin/elm-make which is not added to the PATH, but is accessible by npm run commands — so the issue might be in the gulp command: it's trying to access elm-make from the PATH, not from the node_modules/.bin.
I'll try to find a workaround for that.

@cuducos, @gomex and I are working on it, we found the solution.

I created 2 docker image, one for python and another onde to node (It is hard/right? way, but let's try it)
https://github.com/gomex/jarbas
To reproduce, use these commands:
docker build -t gomex/jarbas-elm -f Dockerfile-elm .
docker run -it --name jarbas-elm gomex/jarbas-elm
docker run -it -p 8001:8001 --volumes-from jarbas-elm gomex/serenata-jarbas

The first command will build the assets
The second will copy the entire jarbas folder to /assets folder (this folder was configured to be accessible from another container)
The third command will run this:
cp -r /assets/jarbas/* /code/jarbas/ &&  python manage.py collectstatic &&  gunicorn jarbas.wsgi:application --reload --bind <IP_ADDRESS>:8001 --workers 4

But the page is still blank and I don't know why :(

@gomex the route / is blank, but /api has content.
Guys, I rewrote the  commit of the @gomex and adapted to stay more easy to develop.
I updated the Dockerfile, Dockerfile-elm and added a docker-compose.yaml.
@cuducos I added a new line in urls.py to serve static files only in development.
My changes are here: https://github.com/gwmoura/jarbas/commit/c30fb061a2a58837abd55dab37b466d1584da545
To test follow this:
docker-compose run --rm elm # this command compile  the elm and generate the app.js
docker-compose run --rm web python manage.py migrate # runs the migrations
docker-compose up web # lifts up the api app listening on 8001 port

Accessing http://localhost:8001/api/ I am see this:

@gwmoura why did you removed postgres installation? I guess it is needed.
In the elm image,
"RUN mkdir /code" it is not necessary.
If you don't copy the code to elm image on the build, you should just run the npm -i after start the container, because node_modules folder won't present on container, it is because when you mount on /code folder, that content created by image on that folder will be overwritten on that container.

This is probably because Django is not prepared to serve static files. Just added that in 23cc355 (temporary). Also did some minor changes in the Dockerfile ( 040687b and 2db9a25). It worked here, can you confirm that was the case? I'm sleepy and I might have overlooked some step unintentionally…

@cuducos I tested here... Still blank :(

@gomex

You tested the route /api? On the home page, is still blank, because the template home - jarbas/frontend/templates/frontend/home.html no has content inside body tag, has only a js scripts

I removed the postgres, because sqlite was defined as default database in development env - https://github.com/datasciencebr/jarbas/pull/47

@gomex and I just paired for a while. The code in extract-nodejs worked here (/static/app.js returned 200 so the page is not blank anymore). We drafted todo list and described the pipeline we have in mind:
TODO:

[ ] Setup nginx
[ ] Setup PostgreSQL
[ ]  Setup timezone within the container

Pipeline:

Build jarbas-elm
Run jarbas-elm (generate jarbas/frontend/static/app.js)
Build & run jarbas-postgres
Build jarbas-python (copying volume from jarbas-elm)
Run jarbas-python (run migrations, generate staticfiles/ & start the app)
Build jarbas-nginx (copying staticfiles from jarbas-python)
Run jarbas-nginx

@gomex @cuducos, I has some updates (https://github.com/gwmoura/jarbas/commit/449afa4deb728c4e3b92130dea7da69b0a8b9528), based in what @gomex talked.
On my commits has:

Build jarbas-elm - service elm on docker-compose.yaml
Run jarbas-elm (generate jarbas/frontend/static/app.js) - docker-compose run --rm elm

Tks @gwmoura — I'm not sure how's the best way to integrate that, so I'll leave it to you and @gomex — fell free to send PRs to extract-nodejs (should I rename it as docker?) and I'll merge it quickly ; )

I made some changes and applied on that PR: https://github.com/datasciencebr/jarbas/pull/48/files

@cuducos @gomex the PR #48 is working fine for me :smile:
@cuducos we can work on extract-nodejs, no problems.

Many thanks @gomes and @gwmoura! Great PR!
I merged it to extract-nodejs but thinking ahead I'd like to raise three questions concerning this PR:

While developing having control of the envvars is very important: which is the way to do it with this Docker setup (without end up adding them to docker-compose.yml or Dockerfile and accidentally committing them)?
I usually avoid Makefile commands on docs because I was told it's not Windows friendly — any opinion on that? I don't have Windows so I have no idea about what to think about it…
I ran into an unexpected problem, I need to clarify this before commiting in case more collaborators end up in the same hole — this issue is detailed below.

I ran into this issue here, not sure why — http://localhost:80/ gives me the following error (an equivalent of a 404 masked because we are running the app in debug mode):
ImportError at /
No module named 'jarbas.frontend.views'
Request Method:	GET
Request URL:	http://localhost/
Django Version:	1.10.3
Exception Type:	ImportError
Exception Value:
No module named 'jarbas.frontend.views'
Exception Location:	/code/jarbas/urls.py in <module>, line 21
…

Or, from docker logs:
Internal Server Error: /
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/site-packages/django/core/handlers/exception.py", line 39, in inner
    response = get_response(request)
  File "/usr/local/lib/python3.5/site-packages/django/core/handlers/base.py", line 172, in _get_response
    resolver_match = resolver.resolve(request.path_info)
  File "/usr/local/lib/python3.5/site-packages/django/urls/resolvers.py", line 270, in resolve
    for pattern in self.url_patterns:
  File "/usr/local/lib/python3.5/site-packages/django/utils/functional.py", line 35, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/usr/local/lib/python3.5/site-packages/django/urls/resolvers.py", line 313, in url_patterns
    patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module)
  File "/usr/local/lib/python3.5/site-packages/django/utils/functional.py", line 35, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/usr/local/lib/python3.5/site-packages/django/urls/resolvers.py", line 306, in urlconf_module
    return import_module(self.urlconf_name)
  File "/usr/local/lib/python3.5/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 986, in _gcd_import
  File "<frozen importlib._bootstrap>", line 969, in _find_and_load
  File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 673, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 665, in exec_module
  File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
  File "/code/jarbas/urls.py", line 21, in <module>
    from jarbas.frontend.views import home
ImportError: No module named 'jarbas.frontend.views'

This is bizarre, because of three reasons:
Reason 1
If I import it through the container, it works:
$ docker run -it --rm jarbas_jarbas python
Python 3.5.2 (default, Nov 10 2016, 08:25:20)
[GCC 4.9.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from jarbas.frontend.views import home
>>>

Reason 2
The tests pass (and yes, this endpoint is tested):
docker run -it --rm jarbas_jarbas python manage.py test
Creating test database for alias 'default'...
.........................................................
----------------------------------------------------------------------
Ran 57 tests in 0.385s

OK
Destroying test database for alias 'default'...

Reason 3
Running the same code locally (not through Docker also works).

Any ideas what did I got wrong here?
AFAIK I deleted all previous Docker images and containers to start form scratch (docker ps -aq | xargs docker rm & docker images -aq | xargs docker rmi) — but ended up the same.

@cuducos how do you ran? You ran using docker-compose up -d?

While developing having control of the envvars is very important: which is the way to do it with this Docker setup (without end up adding them to docker-compose.yml or Dockerfile and accidentally committing them)?

We can configure the docker-compose.yaml to use the host's ENV vars

I usually avoid Makefile commands on docs because I was told it's not Windows friendly — any opinion on that? I don't have Windows so I have no idea about what to think about it…

The Makefile is a shortcut to docker-compose commands, Windows users can run docker-compose commands or maybe we can write a .bat script
@gomex What do you think?

Hey @cuducos
Can you please tell me how can I reproduce that? Did you use make or just docker-compose?

Hey @cuducos I found the problem and solved on that PR: https://github.com/datasciencebr/jarbas/pull/49

While developing having control of the envvars is very important: which is the way to do it with this Docker setup (without end up adding them to docker-compose.yml or Dockerfile and accidentally committing them)?

We copy entire code from your folder, your .env file will be transfered to docker image too.

I usually avoid Makefile commands on docs because I was told it's not Windows friendly — any opinion on that? I don't have Windows so I have no idea about what to think about it…

You are right. We can remove Makefile and just user docker-compose commands :(

I ran into an unexpected problem, I need to clarify this before commiting in case more collaborators end up in the same hole — this issue is detailed below.

I already fixed this problem: https://github.com/datasciencebr/jarbas/pull/49

Many thanks, @gomex!
Got it about teh envvars, tks. I'm testing the PR and probably gonna merge it soon. About the Makefile we can keep it there, but change the docs: we prioritize whatever works for everyone (and maybe in a parenthesis we say something as we have a Makefile to make all that simpler if you like) — my focus is on keeping the focus of the documentation on the instructions that work more widely ; )

I'm closing this issue since it was an open ended discussion (Docker? Travis? Ansible? Capistrano? Fabric?) and now that we have Docker stuff on master we can focus on automating deploy and provision with Docker (just opened a fresh new issue for that).
Many many thanks for all of you that coded to get us further — specially @gomex @gwmoura @luiz-simples @pedrommone ; )


Reactive Variable Support Like Apollo Client 3

does it support reactive variable like apollo client : https://www.apollographql.com/docs/react/local-state/reactive-variables/

Yeah, but I do not think this will happen any time soon. but ofc PR are welcome


kdevelop: 5.0.2 -> 5.0.3

Motivation for this change
New minor version.
Added konsole dependency to make the integrated terminal work.
Things done

[x] Tested using sandboxing
(nix.useSandbox on NixOS,
or option build-use-sandbox in nix.conf
on non-NixOS)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] Linux

[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[x] Fits CONTRIBUTING.md.

Thank you

Hi, can you please merge to 16.09?


Component hydration fails

With latest template using elderjs v1 loading of hydrated component fails
s.min.js:1 Uncaught (in promise) Error: svelte/entryClock-2e8914b9.js, http://localhost:5000/ (SystemJS https://git.io/JvFET#8) at s.min.js:1

Reason is backslash in beginning of import path in index.html
...
System.import('\svelte/entryClock-2e8914b9.js').then(({ default: App }) => {
...

Probably a Windows specific quirck.

@meigo pushed a fix in v1.0.4.
Thanks.

Beautiful, thank you.