增加cpu可以支持的运行 add a cpu version of training-api

    if device == 'cuda':
        model = MMDataParallel(model.cuda(cfg.gpu_ids[0]), device_ids=cfg.gpu_ids)
    elif device == 'cpu':
        model = model.cpu()
    else:
        raise ValueError(F'unsupported device name {device}.')

Thanks for your contribution and we appreciate it a lot. The following instructions would make your pull request more healthy and more easily get feedback. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers.
Motivation
在mmcls的api中可以使用cpu训练，但mmdet不行。就是因为少了这几行代码，故再此处添加。
Modification
在train_detector函数添加传参device，并且加入判断，看是否是cpu。从而决定模型在何处训练。
这样，默认device为cuda，原有的工程不会受影响，而增加了对cpu训练的支持。
Checklist

Pre-commit or other linting tools are used to fix the potential lint issues.
The modification is covered by complete unit tests. If not, please add more unit test to ensure the correctness.
If the modification has potential influence on downstream projects, this PR should be tested with downstream projects, like MMDet or MMCls.
The documentation has been modified accordingly, like docstring or example tutorials.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

https://github.com/open-mmlab/mmdetection/pull/7016 supports CPU training.
The PR has already been merged into the dev branch.

though #7016 has support cpu training, the style is different from the https://github.com/open-mmlab/mmclassification/tree/dev/mmcls/apis/train.py , so I recommand to use the same style like "add a perameter device", and I think it is easier to understand for api users.
Besides, I recommand to uniform all mmlab api code to the same style to support cpu training.

@EasonQYS Maybe we can discuss which way of writing is the best and choose the best.

[question] As for #7016  , if I want to use mmdet/apis/train.py , how to understand "device_ids=cfg.gpu_ids"?
What should be equel to cfg.gpu_ids?
As I know, cfg.gpu_ids should be a non-negtive sequence like [0,] or [0,1,]. but not accept [-1,] or -1.

Can you train a model with your modification? If you do not use the MMDataParallel, how can you deal with the data wrapped in the DataContainer?

Can you train a model with your modification? If you do not use the MMDataParallel, how can you deal with the data wrapped in the DataContainer?

Thank you to remind me, and I have fix if by check mmcv.version.
In older version, I think we don't need to put data to parallel for cpu training.

Can you train a model with your modification? If you do not use the MMDataParallel, how can you deal with the data wrapped in the DataContainer?

Thank you to remind me, and I have fix if by check mmcv.version. In older version, I think we don't need to put data to parallel for cpu training.

Have you tried the training progress with CPU? With the below line, MMDet can not deal with the DataContainer, we must use MMDP to deal with the data wrapped in the DataContainer or there will be an error!
model = model.cpu()

Moreover, MMCls will modify the code according to our current version, so this PR is suggested to be closed.

Thanks @EasonQYS  for your kind contribution. However, model.cpu() is not applicable to make mmdet models run on CPU, it still need MMDP to deal with datacontainer. Therefore, this PR will be closed.


Sounds for entering/leaving ...

Any ideas for that? I might have some funny noises in mind. More "techy" or some kind of warp sound?

I honestly have not thought about it yet and am open for anything ¯_(ツ)_/¯
It probably should be a sound that's also "reversible" so entering and leaving sound, uhm, opposite.
Same for entity shrinking/regrowing.
Crafting only needs one sound.

Sure, give me some time and I'll try my best!

No hurry, this issue is two years old and it had no sound at all yet ;)
Thank you!

Just came with something up while playing with synthesizers.
https://soundcloud.com/sven-kha/sets/cm-1/s-rhZBY

They are quite high pitched indeed. Less highs, more middles and i'd be happy :)

Sure :smile:


Clarification of pairing instructions

Hi. I'm having a problem pairing to a Somfy motorised blind, Basically I don't understand what the first line of the pairing instructions wants me to do. It says "Open the shade motor memory with an existing remote". I'm using a single channel Somfy remote - Situo RTS. I've followed the instructions as far as pairing and have tuned the receiver. I have an RF Explorer spectrum analyser and can see that both the remote and RTS RF module are transmitting at approximately the same level and frequency. I'm also using the latest 1.4.6 update. Any help would be much appreciated and thanks for a great project.

Thanks for the quick reply. I've been pressing the prog button on the back of the remote and the battery powered Somfy motor responds by jogging and the switch on the motor starts to flash green. When I press the prog button within the app, the Somfy motor does not respond. The RF module is about 2 metres from the motor, I'm using Dupont leads to connect the RF module to the ESP32 mounted on a bread board and am wondering if they are too long at about 20 cms although I don't see any corruption receiving the data from the remote. I don't have another RF module to try as they are still on the boat. So I may move on to the tidied up version with shorter wiring.

Double check your wiring.  GDO0 (pin 3 on the transceiver) should be connected to the TX Pin.  If you followed the guide that will be GPIO 13.  GDO2 should be connected to RX (GPIO 12) or not connected at all.  If you omit this wire you simply set RX and TX to GPIO 13 or 12 depending on where you wired it.  The transceiver can receive from either pin and the software controls which mode it operates in.

Yeah I need to update the pictures.  You can build it either way and is just so happens that the picture was built differently than the table.

Unfortunately I now have another problem. I have three blinds all bought at the same time. All three are outwardly identical with the same Somfy battery powered motor. Two of them are now paired but I can't get the third to pair. The motor responds to the single channel remote in the same way as the other two and it goes into pairing mode using the remote's prog button but it won't respond and pair to the ESP Somfy. Perhaps I should open a new issue?

Is this third motor fully charged?  Also, try to increment the address by 10 and see if it will pair.  If it does then something occurred during the pairing process and the address ended up getting reserved in the motor.  If this happens with a remote you need to reset the motor to factory settings and start over but in this instance, you can simply change the address.

Yes, all the motors were fully charged. Anyway, changing the address by 10 allowed the blind to pair so something must have gone wrong with the original pairing attempt. I plan to factory reset the motor and start again from scratch. Thanks for your support, much appreciated.

Closing this for now.  Open a new issue if you have further problems.


After upgrading to 1.9 apple requires strings for location

After upgrading to Flutter 1.9, and with no other upgrades on any packages, I get this message from Apple:

Dear Developer,    We identified one or more issues with a recent delivery   for your app, "Ianum" 1.1.2 (131).
Your delivery was successful, but you may wish to correct the following issues in your next delivery:                        	I
TMS-90683: Missing Purpose String in Info.plist -  Your app's  code references one or more APIs that access sensitive user data. The  app's Info.plist file should contain a NSLocationAlwaysUsageDescription  key with a user-facing purpose string explaining clearly and completely  why your app needs the data. Starting Spring 2019, all apps submitted  to the App Store that access user data are required to include a purpose  string. If you're using external libraries or SDKs, they may reference  APIs that require a purpose string. While your app might not use these  APIs, a purpose string is still required. You can contact the developer  of the library or SDK and request they release a version of their code  that doesn't contain the APIs. Learn more (https://developer.apple.com/documentation/uikit/core_app/protecting_the_user_s_privacy).
ITMS-90683: Missing Purpose String in Info.plist -  Your app's  code references one or more APIs that access sensitive user data. The  app's Info.plist file should contain a NSLocationWhenInUseUsageDescription  key with a user-facing purpose string explaining clearly and completely  why your app needs the data. Starting Spring 2019, all apps submitted  to the App Store that access user data are required to include a purpose  string. If you're using external libraries or SDKs, they may reference  APIs that require a purpose string. While your app might not use these  APIs, a purpose string is still required. You can contact the developer  of the library or SDK and request they release a version of their code  that doesn't contain the APIs. Learn more (https://developer.apple.com/documentation/uikit/core_app/protecting_the_user_s_privacy).
After you’ve corrected the issues, you can use Xcode or Application Loader to upload a new binary to App Store Connect.
Best regards, The App Store Team

[✓] Flutter (Channel beta, v1.9.1+hotfix.2, on Mac OS X 10.14.5 18F203, locale en-PT)

[✓] Android toolchain - develop for Android devices (Android SDK version 29.0.0)
[✓] Xcode - develop for iOS and macOS (Xcode 10.2.1)
[✓] Android Studio (version 3.4)
[✓] VS Code (version 1.38.0)
[✓] Connected device (1 available)

• No issues found!

pubspec.yaml:
environment:
  sdk: ">=2.2.2 <3.0.0"

dependencies:
  flutter:
    sdk: flutter
  flutter_localizations:
    sdk: flutter
  built_collection: ^4.1.0
  built_value: ^6.3.0
  rxdart: ^0.22.0
  pull_to_refresh: ^1.4.6

  #UI
  country_pickers: ^1.0.1
  modal_progress_hud: ^0.1.2
  fast_qr_reader_view: # local
  flutter_page_indicator: ^0.0.3
  transparent_image: ^0.1.0
  flutter_markdown: ^0.2.0
  cached_network_image: ^0.5.1
  flare_flutter:  ^1.5.0
  keyboard_actions: ^1.0.3
  flutter_platform_widgets: ^0.12.0

  # Core
  shared_preferences: ^0.4.3

  # App Linking
  uni_links: ^0.2.0

  # Network
  dio: 2.1.0
  http: ^0.12.0

  # Firebase
  firebase_messaging: ^5.1.2
  firebase_dynamic_links: ^0.4.0+4
  firebase_performance: ^0.3.0+4

  qr_code_scanner:
    path: /Users/goncalopalma/FlutterProjects/openSource/qr_code_scanner

  #translations
  intl: ^0.15.7
  intl_translation: ^0.17.0

  # Misc
  flutter_local_notifications: ^0.8.0
  url_launcher: ^4.0.2
  package_info: ^0.4.0+4
  notification_permissions: # local
  clock: ^1.0.1
  launch_review: ^2.0.0
  connectivity: ^0.4.3+2
  sms_autofill: ^1.1.0

  flutter_crashlytics: ^0.2.0

  cupertino_icons: ^0.1.2

  pedantic: ^1.7.0

dev_dependencies:
  build_runner: ^1.2.3
  built_value_generator: ^6.3.0
  flutter_test:
    sdk: flutter

  flutter_driver:
    sdk: flutter

flutter:
  assets:
  - assets/
  uses-material-design: true

UPDATE:
the connectivity package does seem to have a NSLocation String that I wasn't aware of, but then again, I've been using this package for some months now, with no problem, and didn't update it.
There is a helper method provided in this plugin to request the location authorization: `requestLocationServiceAuthorization`.
To request location authorization, make sure to add the following keys to your _Info.plist_ file, located in `<project root>/ios/Runner/Info.plist`:

* `NSLocationAlwaysAndWhenInUseUsageDescription` - describe why the app needs access to the user’s location information all the time (foreground and background). This is called _Privacy - Location Always and When In Use Usage Description_ in the visual editor.
* `NSLocationWhenInUseUsageDescription` - describe why the app needs access to the user’s location information when the app is running in the foreground. This is called _Privacy - Location When In Use Usage Description_ in the visual editor.

However  the functionality depicted above is not in use in the app

@BondarenkoStas
As far as I'm telling, the following code must be removed (and all the code dependent from it):
  Future<String> getWifiName() async {
    String wifiName = await methodChannel.invokeMethod<String>('wifiName');
    // as Android might return <unknown ssid>, uniforming result
    // our iOS implementation will return null
    if (wifiName == '<unknown ssid>') wifiName = null;
    return wifiName;
  }

  Future<String> getWifiBSSID() async {
    return await methodChannel.invokeMethod<String>('wifiBSSID');
  }

  Future<String> getWifiIP() async {
    return await methodChannel.invokeMethod<String>('wifiIPAddress');
  }

Thank you!
I'll be closing this issue now


Document TM management

The PR is split into two commits:

Various improvements to make the Team / Project / Localization pages doc more coherent.
The actual documentation of the TM management.

I'll add a screenshot when the feature actually lands and the Upload .TMX button is enabled.

@bcolsson I would appreciate your review in particular, since you've worked on Pontoon docs recently.

Is this comment accurate? I'm fairly certain you can localize from this page.

Sure, but this is what we mean by the Localization page:
https://mozilla-l10n.github.io/localizer-documentation/tools/pontoon/teams_projects.html#localization-page

I'll add a screenshot when the feature actually lands and the Upload .TMX button is enabled.

Just to confirm, this is expected to happen in another PR?

Just to confirm, this is expected to happen in another PR?

Yes.


Changed safety approach to distinguish between install and development

Split the safety runs between requirements file for installation and minimum-constraints file for development. The safety run for installation must succeed, while the safety run for development shows its issues, but is allowed to fail.

coverage: 37.029%. remained the same
when pulling 917d4afee470246b8fe17c05a1194c1e394d5819 on anil/make-safety
into 896514feee3d87509718d03b104bec34e60c6f70 on master.


No custom user agent

some times the code running inside electron is generic, and may need access to real userAgent. This small change adds an option to not user custom user agent.
I have tested that it works for our scenario. Please let me know what you think about adding it to main branch.

@twolfson, Can you please take a look at this PR and let me know what you think about it?

Sorry for the delay. On my way back from a very long vacation. I'll review
this within the next 2 weeks
On Wed, May 1, 2019, 11:56 PM Makarand Patwardhan<EMAIL_ADDRESS>wrote:

@twolfson https://github.com/twolfson, Can you please take a look at
this PR and let me know what you think about it?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/twolfson/karma-electron/pull/39#issuecomment-488445003,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAG4KWDVL3GUZAFIE5GHK6DPTIGZ7ANCNFSM4HJRDYGQ
.

This has been released in 6.1.0. Thanks for your help!


Live queries / change feeds / server push / streaming / "realtime"

Copied from an email:
I'd suggest a feature for some next version of arangoDB. The introduced async execution seems a nice idea. Is there a plan to implement websockets too? It would be a good extension of async execution, and eliminate the continuous polling from the client side (simplify the code on client side, and means less server and network overhead).

+1
Especially the combination web sockets and db triggers sounds charming to me :)

Maybe Meteor's DDP protocol could be used for socket connections?
Otherwise, HTTP2/SPDY should do too (server-push).

+1
@phal0r exactly, i guess when database triggers is implimented they can only run a javascript function but with websocket supported by ArangoDB other drivers can use triggers too and have a realtime changes at clients.

Having read a few articles about the topic made me wonder which of the (future) APIs and protocols could be utilized and how.

Server Push (HTTP2/SPDY): servers can push files to the client before the client parsed the HTML and figured out what other resources to request. This is really useful for serving applications and their assets (shorter loading time, no real need to join assets together anymore to reduce number of requests). Should be utilized by Foxx. But not of any use for serving REST requests if I understand correctly (it's not documents in the sense of files that could be pushed, but streamed[?] JSON).

Header compression / multiplexing (HTTP2/SPDY): every HTTP request served using HTTP2 would benefit from smaller overhead. Should cause a little less traffic and allow for lower latency (claimed to require less re-transmits and avoid TCP connection congestion because of lower amount of packets [but only if payload is small?]).

No Head-of-Line blocking (HTTP2/SPDY): Faster transfers? But limited effect, because JSON response can only be parsed from first to last packet.

Binary transmission (HTTP2/SPDY): Faster for the server to send JSON response? (Not if it needs to turn internal data structures to JSON anyway I suppose). No clue if there's any benefit for the client, but I can't see one - JSON is a string that needs to be parsed either way, and it's "binary" already, isn't it?

Server-sent events: Server can inform client about changes in a non-blocking fashion (MIME text/event-stream). Not useful for REST requests, because the client already "knows" about query results that will arrive after a query. Unless a query is known to take a long time to process and you don't want a pending connection in the meanwhile - but then there would still be the event-stream connection to inform the client when the data is ready - dah! (In case of parallel queries by the same client, there could be a single event-stream though instead of one pending connection per request, to notify the client about finished queries).
It's a different story if ArangoDB supported triggers or reactive/live queries however: a client would subscribe a query at the server, and the server notify the client about changes that affect the result of that query (also probably also send the new result, a partially updated document or a delta to patch the client state). I bet this is tricky as hell to implement on the server-side, but really sweat for realtime applications. It's RethinkDB's killer feature.
Not sure how much load it would be on the server and client in case of massive aggregations, that may change every time a new document is inserted (and how to get that to the client efficiently - granted it's more than just a count of how many documented are in the collection). Maybe not really worth to support certain complex queries? It would probably be possible to do some kind of incremental aggregation, but it would require a lot space (e.g. the hash table of a hash-based aggregation must be kept around to incorporate new documents, as well as at least the document ids of all aggregated documents and in which category the fell; not to mention updates and deletions).
I'm not sure how well event-streams handle large payloads. I don't think anything but an update notification should be transmitted, because of the service worker that processes the incoming data (at least in browsers) - except for partial updates, which would usually be small JSON documents or diffs.

WebSockets: More flexible than event-streams, because it's fully duplex and there's no protocol to adhere to unless you want so. A bi-directional communication would allow the client to signal the server that it wants to cancel a query or transmission at any time - to save server load (e.g. query took too long and result is expected to be obsolete, or not needed any longer). Small packet overhead (even less than HTTP2 headers?), which should make data transfer faster in case of large payloads (=many packets). Client needs to implement socket connectivity, but shouldn't be hard if payload is always JSON, and because it's TCP based (no need to handle transmission errors like with UDP).
It would require some sort of header still, to carry the some information that an HTTP header would contain otherwise - such as a status code. Sockets may therefore be less efficient than the HTTP2 transfer method actually, which comes with native header compression. This applies to regular REST requests.
In case of live queries (see above), they may have an advantage over HTTP2 - both, the client and the server, can sync state over a single connection with web sockets as changes  occur on either end. On the other hand, if I understand HTTP2 correctly, then streams are possible there too. The only difference might be that connections undergo different states (WebSockets always open until closed permanently?).

Preliminary conclusion: HTTP2 appears to be the best option out of HTTP2, SPDY, SSE / event-streams and WebSockets. The differences in speed/overhead between HTTP2 and WebSockets seem marginal, HTTP2 might be easier to support by clients however. Browsers as REST clients support HTTP2 automatically (if they support HTTP2) - or is there anything that needs to be implemented on client-side to send and receive data streams? Are multiple streams for a single request even possible with the server in mind? (one result batch per stream, in N parallel streams?)

Isn't with HTTP2/SPDY still all that handshake stuff on every connection ?
Therefore i would still prefer websockets.
Also flexibility is much higher with complete bi-directional connections.
Like server shedules, connection alive checks, etc.
CoDEmanX<EMAIL_ADDRESS>schrieb am Mi., 9. Sep. 2015 um
12:40 Uhr:

Having read a few articles about the topic made me wonder which of the
(future) APIs and protocols could be utilized and how.

Server Push (HTTP2/SPDY): servers can push files to the client
before the client parsed the HTML and figured out what other resources to
request. This is really useful for serving applications and their assets
(shorter loading time, no real need to join assets together anymore to
reduce number of requests). Should be utilized by Foxx. But not of any use
for serving REST requests if I understand correctly (it's not documents in
the sense of files that could be pushed, but streamed[?] JSON).
Header compression / multiplexing (HTTP2/SPDY): every HTTP request
served using HTTP2 would benefit from smaller overhead. Should cause a
little less traffic and allow for lower latency (claimed to require less
re-transmits and avoid TCP connection congestion because of lower amount of
packets [but only if payload is small?]).
No Head-of-Line blocking (HTTP2/SPDY): Faster transfers? But limited
effect, because JSON response can only be parsed from first to last packet.
Binary transmission (HTTP2/SPDY): Faster for the server to send JSON
response? (Not if it needs to turn internal data structures to JSON anyway
I suppose). No clue if there's any benefit for the client, but I can't see
one - JSON is a string that needs to be parsed either way, and it's
"binary" already, isn't it?
Server-sent events
https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events:
Server can inform client about changes in a non-blocking fashion (MIME
text/event-stream). Not useful for REST requests, because the client
already "knows" about query results that will arrive after a query. Unless
a query is known to take a long time to process and you don't want a
pending connection in the meanwhile - but then there would still be the
event-stream connection to inform the client when the data is ready - dah!
(In case of parallel queries by the same client, there could be a single
event-stream though instead of one pending connection per request, to
notify the client about finished queries).
It's a different story if ArangoDB supported triggers or reactive/live
queries however: a client would subscribe a query at the server, and the
server notify the client about changes that affect the result of that query
(also probably also send the new result, a partially updated document or a
delta to patch the client state). I bet this is tricky as hell to implement
on the server-side, but really sweat for realtime applications. It's
RethinkDB's killer feature.
Not sure how much load it would be on the server and client in case of
massive aggregations, that may change every time a new document is inserted
(and how to get that to the client efficiently - granted it's more than
just a count of how many documented are in the collection). Maybe not
really worth to support certain complex queries? It would probably be
possible to do some kind of incremental aggregation, but it would require a
lot space (e.g. the hash table of a hash-based aggregation must be kept
around to incorporate new documents, as well as at least the document ids
of all aggregated documents and in which category the fell; not to mention
updates and deletions).
I'm not sure how well event-streams handle large payloads. I don't
think anything but an update notification should be transmitted,
because of the service worker that processes the incoming data (at least in
browsers) - except for partial updates, which would usually be small JSON
documents or diffs.
WebSockets: More flexible than event-streams, because it's fully
duplex and there's no protocol to adhere to unless you want so. A
bi-directional communication would allow the client to signal the server
that it wants to cancel a query or transmission at any time - to save
server load (e.g. query took too long and result is expected to be
obsolete, or not needed any longer). Small packet overhead (even less than
HTTP2 headers?), which should make data transfer faster in case of large
payloads (=many packets). Client needs to implement socket connectivity,
but shouldn't be hard if payload is always JSON, and because it's TCP based
(no need to handle transmission errors like with UDP).
It would require some sort of header still, to carry the some
information that an HTTP header would contain otherwise - such as a status
code. Sockets may therefore be less efficient than the HTTP2 transfer
method actually, which comes with native header compression. This applies
to regular REST requests.
In case of live queries (see above), they may have an advantage over
HTTP2 - both, the client and the server, can sync state over a single
connection with web sockets as changes occur on either end. On the other
hand, if I understand HTTP2
https://http2.github.io/http2-spec/#StreamsLayer correctly, then
streams are possible there too. The only difference might be that
connections undergo different states (WebSockets always open until closed
permanently?).
Preliminary conclusion: HTTP2 appears to be the best option out of
HTTP2, SPDY, SSE / event-streams and WebSockets. The differences in
speed/overhead between HTTP2 and WebSockets seem marginal, HTTP2 might be
easier to support by clients however. Browsers as REST clients support
HTTP2 automatically (if they support HTTP2) - or is there anything that
needs to be implemented on client-side to send and receive data streams?
Are multiple streams for a single request even possible with the server in
mind? (one result batch per stream, in N parallel streams?)
—
Reply to this email directly or view it on GitHub
https://github.com/arangodb/arangodb/issues/602#issuecomment-138869059.

Both are TCP based, hence both require handshaking. Not sure about about connection alive checks, is it even needed for HTTP? (there's keep-alive)
HTTP2's benefit over HTTP1.x is that multiple, independent streams of data (e.g. different files) can be transferred through a single TCP connection. In HTTP1.x, it can only be a single stream / file, thus browsers open 4-8 TCP connections (each needs hand-shaking AFAIK) for parallel transfers. More connections would cause even more overhead. HTTP2 and WebSockets should be on par regarding protocol overhead, because they can transfer any data, interleaved, using a single connection.
To be honest, I don't see any difference in flexibility between both, given that we need the HTTP header anyway. The payload would be transferred in the same way, only the way the meta information (header) is encoded would differ. Encoding and decoding it in case of WebSockets would need custom code, whereas there's broad support to set and parse HTTP headers in existing tools and frameworks.
HTTP2 is just as bi-directional from a technical point of view (the server needs to be programmed to react on responses while answering of course - like for killing queries/transfers right in the middle of it) - if I'm not mistaken.
Can you elaborate on server schedules?

I meant with websockets - you only have handshake mechanism on establishing
the connection.
That's still an opportunity over any HTTP connection regarding connsecutive
calls.
And when you have a stable connection, you know about a device is
disconnecting/reconnecting (and can do local storage/securing/closing stuff
on both sides) - that's what i menat with alive chechks.
That for me are all points for websocket connections.
Greets
CoDEmanX<EMAIL_ADDRESS>schrieb am Mi., 9. Sep. 2015 um
13:35 Uhr:

Both are TCP based, hence both require handshaking. Not sure about about
connection alive checks, is it even needed for HTTP? (there's keep-alive)
HTTP2's benefit over HTTP1.x is that multiple, independent streams of data
(e.g. different files) can be transferred through a single TCP connection.
In HTTP1.x, it can only be a single stream / file, thus browsers open 4-8
TCP connections (each needs hand-shaking AFAIK) for parallel transfers.
More connections would cause even more overhead. HTTP2 and WebSockets
should be on par regarding protocol overhead, because they can transfer any
data, interleaved, using a single connection.
To be honest, I don't see any difference in flexibility between both,
given that we need the HTTP header anyway. The payload would be transferred
in the same way, only the way the meta information (header) is encoded
would differ. Encoding and decoding it in case of WebSockets would need
custom code, whereas there's broad support to set and parse HTTP headers in
existing tools and frameworks.
HTTP2 is just as bi-directional from a technical point of view (the server
needs to be programmed to react on responses while answering of course -
like for killing queries/transfers right in the middle of it) - if I'm not
mistaken.
Can you elaborate on server schedules?
—
Reply to this email directly or view it on GitHub
https://github.com/arangodb/arangodb/issues/602#issuecomment-138883444.

There's only one handshake on establishing the connection in HTTP communication as well?! It must use keep-alive of course.
Consecutive calls are not a necessity either, see http://stackoverflow.com/questions/23419469/is-http-1-1-full-duplex

And when you have a stable connection, you know about a device is
disconnecting/reconnecting (and can do local storage/securing/closing stuff
on both sides)

Hm, you can't know for sure, unless the client sends a heartbeat in regular intervals to the server or makes requests. Not sure how this would be done in HTTP exactly, but you could probably do about the same to prevent the connection from closing (does it transport layer close it after a timeout, without client or server explicitly closing it???) and inform the server that the client is still there.
Need to investigate further how long-lived HTTP connections can be, but thinking of long polling, it shouldn't be an issue. And keep in mind that you'll usually request data from ArangoDB (little data) and then receive multiple large batches of JSON payload (much data) until the query cursor is exhausted.
There's no real need for low-latency messaging (ping-pong fashion; little data) between client and server. HTTP2 should bring the header overhead down on par with WebSockets, and may even be superior when retrieving one giant JSON response, because WebSockets are message based and not streamed: if the entire JSON response is considered one message, then it will be available to the client not before transfer completed, see: http://stackoverflow.com/questions/14703627/websockets-protocol-vs-http
Another great article: https://devcentral.f5.com/articles/spdy-versus-html5-websockets
Just a thought: WebSockets might be slower in the end compared to HTTP2, because firewalls and anti-virus scanners need to perform deep data inspections for WebSockets - they have no clue what is transferred, whereas HTTP headers give hints.

HTTP2 has built-in support for "heartbeats":
https://httpwg.github.io/specs/rfc7540.html#PING
A major drawback of HTTP2 is that there is not way to detect server pushes in JS (although it's a JS limitation, not HTTP2). It needs to be combined with an event-stream to signal when data has arrived. That's a plus for WebSockets, but who knows what JS will do about HTTP2.

Are there any concrete plans for a websocket implementation? I do hate to ask, so I'm sorry if I seem impatient or something, I've just had my eye on this in the road map for quite some time now :)
Regards.

Currently we don't have an ETA for this. We are working on migrating to velocypack for disk and network IO. Velocypack can produce json, or a binary data format which is compact and fast parseable. This will also be used in the mesosphere integration. After all this is done, we may think about starting out on this.

+1
Pushing database changes to clients would be a big enhancement. I'm working on an app with offline storage capabilities and websockets would allow the server to push updates in real-time to client-side database copies while online.

I'm also very interested in this. Having just recently used socket.io on my mtr.guru side project, I really appreciate the simplicity of it and would love to be able to use it with Foxx. I'd want to use it to push updates to my client side, and also for communicating with a node.js process. There's a socket.io protocol and a client implementation in C++ that might be useful here.
Good recent article on web sockets vs. HTTP/2 & SSE

+1 realtime apps are the thing now

+1 websockets, server pushes, realtime

+1. Lack of this feature is a show-stopper for us

We are discussing the possibilities for this internally.
To give you an example: We could support triggers for collection on which you could react in Foxx services. In addition, we would provide a possibility to push out results to the Foxx service consumers. The triggers could call with an option to configure the time interval how often you would be informed about changes. In this case, you have to handle the changes be yourself. The upside is that such a way can also handle complex queries.

We need the ability for Foxx to use Node's net module.   The following code thrown an exception in Foxx:
var Socket = require('net').Socket;
var s = new Socket({});

We need sockets, to connect to various DB servers from a Foxx job and selectively import the data

@weinberger, while collection triggers wouldn't address our need to establish socket connections with other DB servers (see my comment above), it would nonetheless be a great feature for building modern pub/sub applications that push data to clients and devices without polling.

The simplest thing for subscription could be an http api call
GET /_db/{database-name}/subscribe/{collection-name}/{event types}

event-types
http method equivalent
description

a
-
all events below

c (x)or s
POST
create document

g
GET
read document

p
PATCH
patch a document

r
PUT
put document (replace)

d
DELETE
delete document

Example:
to listen to all documents that are patched in collection testCollection
const subscription = subscribe('http://<IP_ADDRESS>/_db/_system/subscribe/testCollection/p');

subscription.on('p', (doc) => {
  console.log(doc._key); // would print '12345';
});

await arangodb.testCollection.save({_key:'12345'});
await arangodb.testCollection.update({_key:'12345', 'rating':'AAA'});

The server emitted steam is
0000000: 7000 0000 1f7b 225f  p....{"_
0000008: 6b65 7922 3a22 3132  key":"12
0000010: 3334 3522 2c22 7261  345","ra
0000018: 7469 6e67 223a 2241  ting":"A
0000020: 4141 227d            AA"}

first the event type then big endian 4byte unsigned int for the length of the json (0x1f = 31byte) followed by the document.

@baslr that'd be too limited for me. My clients need certain key/values from a filtered join which wouldn't be doable with your suggestion.

@pekeler
maybe introduce a convention j. If the collection is involved in a join.
Example:
const subscription = subscribe('http://<IP_ADDRESS>/_db/_system/subscribe/Persons/uj');

subscription.on('p', (doc) => {
  console.log(doc); // would print the updated docs in collection Persons;
});

arangodb.query(`
FOR p In Persons
  FILTER myfunctions::mustPayTax(p)
  LET distance = LENGTH(
    SHORTEST_PATH(Persons, Friends, p, 'Persons/godfather', 'any'))

  FOR c IN Cities
     FILTER p.zip == c.zip

      UPDATE p WITH {taxPaid: p.taxPaid * c.discount * distance}
        IN Persons
`);

Have you guys considered something like AMQP for triggers / websockets? There is Apache Qpid Proton for example.
Triggers could simply post messages, Foxx controllers (for instance) could be set up to consume and act on them, and basically act as a bridge between websocket connections and AMQP.

For years now, I've been researching for the most flexible realtime pub-sub implementation at database level and so far RethinkDB seems to have the best implementation. However, I cannot get used to the RQL query language in RethinkDB so I'm still looking. I believe the most flexible implementation will be at query language level and I have proposed something for the new ScyllaDB  here. I really like the new ArangoDB v3 release and think something like this as a major feature would really make it stand out.

+1 on doing it the way RethinkDB does.

+1 for live queries

Hey,
for everyone how wants to be notified in realtime when collection / document changes happen: maybe https://github.com/baslr/arangochair is suitable.

@baslr interesting, this way the client is like a node in replication that receive changes

What's the status of this? Any ETA?

-1 for WebSockets. WebSockets isn't a feature description. It's a protocol. I'm not even sure what half of the feedback in this discussion means. Do you mean WebSockets between client and server (Foxx), or server/client and db via the HTTP API?
Also, are you aware of the performance/scalability issues with WebSockets, or that it's incompatible with HTTP/2, or that SSE is a viable option? (at least if you don't need to support MS Edge or use a polyfill until they implement it).
Let's leave the server-client protocol decision to the server devs implementing this, and instead talk about pushing/pulling for changes from the db (triggers, hooks, server to db subscriptions, changefeeds, AMQP, rabbitmq etc).
Personally I don't know half of the technologies I just listed, but I believe triggers is the only one in the roadmap.
Triggers is not my first choice to cover this. I wouldn't want to handle this in the db using something like CREATE TRIGGER ..., but rather subscribe to, or pull for changes. The server would know which information that needs to be kept up to date, while the db wouldn't. Triggers could function as a way to filter from the db what to push to the server, if the alternative is something like pushing every change, but I don't think either alternative is very good or a big improvement over watching the https://docs.arangodb.com/3.0/HTTP/Replications/ReplicationLogger.html.

-1 for WebSockets. WebSockets isn't a feature description. It's a protocol.
I haven't thought about this, but I think @friday is right.
I don't care about the actual protocol the devs choose, but the push functionality.

Perhaps a more reasonable approach would be the ability to set up webhooks for triggers.
More generically to run code when defined events/conditions occur.
Webhooks come to mind because they are fairly simple to implement with defined interfaces.
Github and Gitlab hooks would be a great built in but perhaps as a foxx service.
Sent from my iPhone

On 24 Oct 2016, at 2:09, friday<EMAIL_ADDRESS>wrote:
-1 for WebSockets. WebSockets isn't a feature description. It's a protocol. I'm not even sure what half of the feedback in this discussion means. Do you mean WebSockets between client and server (Foxx), or server/client and db via the HTTP API?
Also, are you aware of the performance/scalability issues with WebSockets, or that it's incompatible with HTTP/2, or that SSE is a viable option? (at least if you don't need to support MS Edge or use a polyfill until they implement it).
Let's leave the server-client protocol decision to the server devs implementing this, and instead talk about pushing/pulling for changes from the db (triggers, hooks, server to db subscriptions, changefeeds, AMQP, rabbitmq etc).
Personally I don't know half of the technologies I just listed, but I believe triggers is the only one in the roadmap.
Triggers is not my first choice to cover this. I wouldn't want to handle this in the db using something like CREATE TRIGGER ..., but rather subscribe to, or pull for changes. The server would know which information that needs to be kept up to date, while the db wouldn't. Triggers could function as a way to filter from the db what to push to the server, if the alternative is something like pushing every change, but I don't think either alternative is very good or a big improvement over watching the https://docs.arangodb.com/3.0/HTTP/Replications/ReplicationLogger.html.
―
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

Looking for an alternative to my rethinkdb stack like tons of others do now when they are bankrupt. If you get live queries like described above im in!

@rnenjoy It would be helpful to understand what exactly you need. Just push notification about insert, update and delete of documents?

https://www.rethinkdb.com/blog/realtime-web/
https://rethinkdb.com/docs/changefeeds/javascript/
Scott

Thanks @smolinari but I know them very well. My question was unclear.
What parts from the current Rethink functionality do you really need?
We will not be able to build all of it in one step and more important I'm not sure if all is necessary.

Hmm.... I can't imagine a partial feature set for live queries. To me it is a simple system, theoretically. A query is set live, the client gets a token for subscribing and then any update, insert or delete, which might alter the query's result is then sent to the client, informing of the change.
Scott

What if your AQL query contains a deep graph traversal followed by complex aggregation? If you would set this to "live" mode, without additional trickery on the server-side the query would need to be rerun on ANY change to the collection to ensure that the result is still correct and if it changed, send the new result to the client (or if you wanted only a diff, how would it be defined and in what format?)

The concept you describe is the OrientDB approach. That has IMHO downsides. If you make every potential query automatically updated, your database can quickly show an unpredictable behavior. That's something nobody wants.
So I see a lot of possible differentiations here. I recently spoke with many Rethink users about it, and the real world requirements differ from use case to use case.
Therefore it would be great to learn a little more about @rnenjoy use case.

I think it's reasonable to consider graphs might require more consideration.
rethinkdb doesn't do graphs.
It's a document store.
Naturally, if a document is particularly large there is a question to the details and what ifs.
Also, arangodb is using documents as the graph model. For a graph query it should be a fast lookup, but you also would not want truly constant notifications usually.
If you did, you'd naturally need to consider what happens on both ends.
The use case/ user story is everything here.
Sent from my iPhone

On 25 Nov 2016, at 19:36, Simran<EMAIL_ADDRESS>wrote:
What if your AQL query contains a deep graph traversal followed by complex aggregation? If you would set this to "live" mode, without additional trickery on the server-side the query would need to be rerun on ANY change to the collection to ensure that the result is still correct and if it changed, send the new result to the client (or if you wanted only a diff, how would it be defined and in what format?)
―
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

@uchuugaka Not just the graphs could be the problem. Even joins can be problematic. There are reasons why also Rethink has limitations here, and they did work on it for a long time.
As more as we learn from your real world use cases as better, we can try to find the right solution.

I've taken the liberty of removing the e-mail quotes from some of the comments where GitHub didn't detect and truncate them automatically. GitHub's markdown parser was reacting quite badly to some of them.

@weinberger I think my use case is pretty simple for the apps i'm making. But i'm not sure it would be easy to implement.
My webapps has queries like:
Get all the chatMessages with userId 1, sort them by date DESC... Subscribe to this query
The user interface updates automatically when new chatMessages to this user arrives since they are pushed to the user
Get all the the sale statistics on outgoing offers to customers from all salesmen. Subscribe to this query so the salesboss gets a live updating UI on the status of all active offers
This is how my use cases are for most of the time.

My use case for such feature would be sending notification to users, live data such as business intelligence reports, and chat.
All this can still be solved by polling an endpoint every second, but after having used Firebase for a year polling seems like a prehistoric thing to do. 😄

@weinberger

If you make every potential query automatically updated, your database can quickly show an unpredictable behavior. That's something nobody wants.

Where is an unpredictable behavior supposed to come from?
More importantly, the potential for any query to be set to "live" has to be in the hands of the developer. If the database is selective about what queries the dev can make live, then that means the database is tying the hands of the developer or rather reducing their ability to get their work done, whatever that work may be. Any hand tying should be avoided more than possibly the database having issues with the query selections.
Scott

A pragmatic approach might be to implement a live-view for Aardvark. When viewing a collection, have an indicator next to the doc-count in the bottom left if there's been any changes on the server since the list was rendered. Implement this without polling.
Step 2: Make this update-notification respect the current filter.
Whatever solution you come up with for this scenario would likely solve my use case.

@smolinari

More importantly, the potential for any query to be set to "live" has to be in the hands of the developer. If the database is selective about what queries the dev can make live, then that means the database is tying the hands of the developer or rather reducing their ability to get their work done, whatever that work may be. Any hand tying should be avoided more than possibly the database having issues with the query selections.

I disagree. If the ArangoDB team make it easy to use their product in inefficient ways, then developers will blame the product and they will fall behind in performance test written by developers who didn't optimized for performance to begin with. Most devs work with deadlines and don't have the time to read the manual, and probably satisfies at the point that their queries work. But will they still work when your app/thingy scales? Thinking about this in beforehand is not the same thing as "hand tying" https://blog.codinghorror.com/falling-into-the-pit-of-success/

Does Arango currently limit the way a developer can query the database, so that the server is always working to its fullest potential? I don't think so. It is the nature of the database beast.
Developers will always blame the database for poor performance, when they think that is where the issue lies. Trying to produce some sort of limit isn't going to stop that. In fact, if you try to tell the developer that "the limits are for their own good, so they can't screw up the performance of the database", they'll probably get upset. Developers are, in almost all cases, quite smart people and limiting them only adds to their frustration.
And you are right. Most developers, most people, don't read manuals, unless they have to.
But, what is worse? Starting with a product thinking you can get somewhere, only finding out it doesn't support what you want to, because that feature is "limited". Or, finding out that you can do what you want, however, you need to do it in a way that doesn't break the database. One way is a dead end. The other is only a fork in the road, both for the dev and the vendor.
If you ask me, putting a limit on a feature in the name of "save the performance", before the feature is even created, is a form of YAGNI.
Scott

You're not being helpful. No one even talked about limiting features. This is a discussion and there is no set solution as I read it. The devs are actively asking for our feedback and use cases in order to solve this as smart as possible by adding the underlying features in order of importance to build a cohesive whole picture. "Implement all the things" is not a good strategy, as proven by some unnamed competitors. You can call the absence of that "hand tying" if you like, but I wouldn't want to use a dbms which is also a photo editing software for example, and even less would I want to support such a product as a developer.
Developer freedom is not about one abstraction layer covering all your use cases. In fact that's kind of the opposite of dev freedom.
Also: Even great developers tend to have colleges, who they share maintenance debt with.

You're not being helpful.

Offering my viewpoint is helping.

No one even talked about limiting features.

Claudius said this:

If you make every potential query automatically updated, your database can quickly show an unpredictable behavior. That's something nobody wants.

Ok. It isn't saying directly that a limitation will be made. But, it is certainly close enough to start the discussion of possible limitations.

"Implement all the things" is not a good strategy, as proven by some unnamed competitors.

Who said implement "all things". I am saying, implement a single feature, live query, and don't worry about "potential unpredictable behaviors" or performance, until such problems arrive.

I wouldn't want to use a dbms which is also a photo editing software for example, and even less would I want to support such a product as a developer.

This is just silly argumentation. It is taking my point of view to an absurd extreme, which is not representative of the true situation. That isn't being helpful.

Developer freedom is not about one abstraction layer covering all your use cases. In fact that's kind of the opposite of dev freedom.

Developer freedom is offering a feature that works in as many use cases as possible. I also understand what it means to work within constraints, which is a bit of a different subject. No software engineer/ designer should purposely add constraints to a feature, before the real problems are even known. That just isn't how software is designed, unless putting a constraint on the user is actually part of the design, like with Twitter.
I just don't see any reason to consider constraints or limits with any new feature, IF the concern is about performance. If Claudius can give some valid reasoning to my first question about unpredictable behavior, then those reasons might be good enough to add some constraints/ limitations. But, then how can he even know that the situation he is possibly designing constraints for is even true?
I follow the motto.

It is better to start imperfectly, than wait for perfection. 😄

Because, there is no such thing as perfection.
As for this discussion, I am done. I've had my say and Claudius and team can do with my suggestion as they please.
Scott

I experimented with some sort of live query thing. You register your query + collections. Every time something changes the query is executed and the results are delivered.
Regarding live the query should only executed in sequence.
The problem I have is how to handle the diff in the previous result and the current result.
Here is an example
chair.liveQuery('for s in scores sort s.score desc limit 5 return s',
['scores'],
{diff:true,initialResultSet:false},
(results) => {
    console.log(results);
});

@smolinari:
You really don't need to subscribe to every potential query (which you also can't in RethinkDB: https://github.com/rethinkdb/rethinkdb/issues/3997). For example you really don't need to subscribe to age calculation like DATE_DIFF(DATE_NOW(), user.birthdate, 'y'). Watching full queries would enable users to do this without even thinking about it (it could be part of a query where some other details needs to be kept updated). And as you are probably aware of DATE_NOW() changes 1000 times per second, unless limited to trigger less often with live queries, but then we have a restriction, and we are still over-subscribing.
@ all:
To prevent over-subscribing, I'd prefer subscriptions to be explicit and based on nodes. You could still do the same thing, but you're less likely to do it accidentally.
Here is an example of this in graphql from a React conf earlier this year about upcoming features in GraphQL. As you can see only likeCount is subscribed to in this query:
{
  feed {
    stories {
      author { name }
      message
      likeCount @live
    }
  }
}

I'm not sure what the AQL version of @live would look like.
I also think subscriptions would be more performant by breaking down queries to start with the nodes being watched. With the likeCount example above that would be starting with watching likes, and filtering based on which new/removed likes are connected to the "story". Since "stories" aren't live it doesn't need to go further (unless I misinterpreted it). The problem is if likeCount is actually calculated with COUNT, because ArangoDB would have to remember partial results from previous queries in a stateful manner or refetch the query again or send the result as differential (+1, -1 etc).
If people want to watch the result of COUNT and other functions I think they should store it.
Also, cred to @baslr for working/experimenting with this. I think it's easier for a community based solution to jump in and fill the gap, because of less pressure to do it perfectly and having to support it later.

Maybe its worth to publish what I've done so far. 🤔

Having the opportunity to just @live on a single field would be awesome.  But i think the most common approach is to watch a filtered collection for new and deleted items.

@baslr: Maybe you could push it to a devel branch? I just wrote a comment (as an issue) at your arangochair github btw.

i am very interested in this feature, i am  literally just getting started with arrango db... I am a high level programmer.......so my ability to write it in C++ is just not there.  but thank you to anyone who ends up writing this feature.

Interesting conversation, and happy to see 'triggers' are on the roadmap (I really need them 👍).
I agree with @friday that we shouldn't be talking about protocols (yet), but about use-cases.
Sorry this reply turned into a long story...
I am looking to use triggers to trigger/alert further actions as soon as document gets created, updated, deleted, etc. I've come up with a couple of use-cases that hopefully help the discussion:
Example use-case 1: a document with field status: 'pending' get's updated to status: 'done'.
As soon as the commit completes, a function is triggered that adds a new field marked_done: '2016-12-31 10:00' and updates the existing field assigned_to to variable value var project_lead.
(workflowy use-case)
Example use-case 2: append author.name to a list-field book.main_authors for all authors that each contributed more than 15 pages to the book (= main authors of a book).
(enrichment use-case)
Example use-case 3: a document is inserted into collection Jobs with fields job_type: 'process_csv', job_blob: 'DEADBEEF1234' and processed: false.
A function is triggered that performs a REST call to an endpoint, signalling job { job_id: 1234 } is ready for processing.
(external interaction use-case)
Example use-case 4: a document is inserted matching a query to do some whacky joining and traversal to generate a certain view (?) on a series of documents.
As soon as a document is inserted, matching the whacky query, the query/function is executed.
E.g.: when a new document is added to the Spells collection, send an updated value total_magic_credits to a message queue endpoint*, by summing field spell_credits for all Spells documents matching player: $Spells.player.name.
(updating view use-case?)
Does this help?
For the devs/interested parties among us: take a look at MarkLogic's reverse query mechanism, which seems to do this very well (and for a long time already):

An alerting application is used to notify users when new content is available that matches a predefined (and usually stored) query. MarkLogic Server includes several infrastructure components that you can use to create alerting applications that have very flexible features and perform and scale to very large numbers of stored queries.
MarkLogic Docs - Creating Alerting Applications

If I'm not mistaken, they serialize a document and see what queries it would match (based on indexes), and then only execute these queries. Also see: cts:reverse-query.
About commits, I could imagine there being interesting use-cases of executing triggers before commit (e.g. lookup city for field postal-code before creating Customer), however, this would probably negatively impact ACID-compliance and/or asynchronicity?
I think it is important to consider the following (IMHO):

trigger != connection: I want to be alerted, but do not want to keep a connection open to the DB
trigger != pub/sub, use nodejs endpoint / message queue / AWS SNS/SQS for that?
stick the query paradigm: trigger query has one part matching some doc(s), other part executing some action(s)?
if the trigger is event based, then the application layer should handle collating events into an (updated) view (send the delta, don't resend the view)?
for external interaction, just signal that a new state/job is available, don't push the full state/payload (among others: reactive manifesto)
don't become a message queue like Kafka? or maybe do?

Happy to hear your comments!
** I would architect use-case 4 as follows: ArangoDB sending trigger message to message queue, message queue subscribed to by a nodejs process, nodejs process serving a websocket per player, browser handling the messages.
This gives you multiple levels of control, offloading of unnecessary load off of the database (your most vulnerable component) and a single websocket for each player's browser to handle both application, database and other events.

We are really in need of changes feeds.
We considered switching to PostgreSQL to use its LISTEN/NOTIFY feature, but at last we decided to use RabbitMQ for changes feeds.

@p30arena Are you at liberty to elaborate on your use case and how change feeds could help with that?

@pluma  Actually, were reinventing the wheel for Instant Messaging.
We need the chat servers to notify users of new messages arriving to a chatroom.
There are a couple of listeners to the count of chat servers; listening to queues based on rooms.
Its not sufficient for us to connect and manage chat servers in a p2p manner for message passing, so we need a broker for message queueing like Kafka, RabbitMQ, etc.
But the simpler and easier to manage solution for us is to harness PUB/SUB features present in some DBMSes.

@pluma: whon do you think about bonefish? it has websocet built on boost asio.
http://wamp-proto.org/ aslo http://autobahn.ws/
I think it may be used asa notification in the asyc requestas to deliver of message, the result of (long) query is done. and may be used to solve of use cases from  @aldegoeij and @p30arena

Maybe db._useDatabase(name).subscribe('change/add/delete/update', cb) for Foxx. This will cover 50% use cases.

@akatsak the repository was  since 3.x removed and subscribe is not available.
with foxx one comes no further, because the notification can not be delivered by the server to the client. the required protocols are missing.  even the SSE (Server-Sent Events) from HTTP/1.1 is not implemted.
to solve of the listed use cases it's necessary to use another tools. the ArangoDB team does not seem to be able to implement that.
https://docs.arangodb.com/3.1/Manual/Foxx/Migrating2x/Repositories.html#triggers
https://html.spec.whatwg.org/multipage/comms.html#server-sent-events
http://blog.arungupta.me/rest-vs-websocket-comparison-benchmarks/

Is there any way to find out about events(add/delete/update) in the document collection using Foxx?

There has not been many updates on this topic since January. Is it still a priority, and if so, is there a roadmap somewhere to follow?

I just spent 2 years on a large project doing this sort of thing with a custom in-house graph database. I ran across this thread because a co-worker couldn't stop saying nice things about ArrangoDB, and I wondered whether it could possibly have been a viable alternative.
We had 2 ways of setting up a "listener":

by the type of an atom/edge (in an RDBMS, this would have been the equivalent of listening to every record in a table)
provide it an initial query. then get back a stream of data containing updates about whichever nodes/edges were part of that query result at the time we issued the query

And then we'd get diffs in a data structure that amounted to {"id": x, "propertyName": y, "value": z "action": "add/remove"}
There were three major missing features:

Just monitor a count of things that satisfy some query
Deal with arbitrary queries which might have their result sets change (say, property "foo" changes to a new value which starts satisfying the WHERE clause, so that now there's a new record to monitor, or it quits being one we care about)
Just being able to specify a subset of the single table version would have been a major win

For our use case (basically, real-time network monitoring), what we had was usable (but just barely) with our custom web socket interface. We talked about switching to HTTP/2, but it never seemed worth the time/energy to figure out ways to make it work.

@jimrthy Arangodb has a handful of stability issues in production.
I hope the team stays focused on these issues instead of new features, we don't want to see a new OrientDB
.

I still don't see any solution to gather real time data. Any update? (I should just get used to PostgreSQL not scaling I guess)

I would also like to see a solution for fetching real time data. Mongo has recently implemented change streams, however I believe realm.io is doing it quite right. It provides fine grained notification updates even for fields changed in 'documents' (they have objects). It would be really nice to see something similar.

Will this ever happen?

This issue has been open and somewhat active for 5 years now, any intention of it going anywhere???

I would love to see this in action. Just thinking about starting to use Arangodb and hit this...
Use case would be to have a trigger for new docs and more importantly, new links (relationships) in the graph.

+1.
This would be very useful.

@Simran-B  I don't think that ArangoDB supports plugins yet. But it would be interesting to see some kind of a streaming solution applied, or even publishing an event to a message queue for external processing. Not sure if the WAL is the best solution for this as mentioned in #8390
Example:
Neo4j Kafka streams allows you to produce or consume streams to update or execute graph queries.
Use Cases:
Producer: based on the topics set up in the Neo4j configuration file. Outputs to said topics will happen when specified node or relationship types change
Consumer: based on the topics set up in the Neo4j configuration file. When events for said topics are picked up, the specified Cypher query for each topic will be executed
Procedure: a direct call in Cypher to publish a given payload to a specified topic
Reference:
https://neo4j-contrib.github.io/neo4j-streams/
https://medium.freecodecamp.org/how-to-embrace-event-driven-graph-analytics-using-neo4j-and-apache-kafka-474c9f405e06

What is the current status of triggers or plugins in arangodb? I will start a project with arangodb soon and coming from neo4j I'm looking for something similar to TransactionEventHandlers

Don't worry guys, nothing will be happening here. this has been outstanding for donkeys of years. I tried implementing it myself but I am not a C++ developer and given I am not a paying arangodb customer I have had to give up and fudge it with Etcd. This single feature will take arangodb straight into the event-sourcing domain. This is exactly what https://axoniq.io/product-overview/axondb is selling for big bucks... but hey we gotta move on

Any update on this issue?

would love for this, will reduce the number of things to have to configure and maintain

++1 for this feature

Any progress?

Sorry to say that but... I'm probably not the only one:
I don't use ArangoDB in production because there is this need. This issue dates from 7 years now. While RethinkDB or CouchDB propose a "changes" API (CouchDB propose several stream types, like continuous or event-stream BTW).
This is probably the only one topic that avoid me to use Arango, it's a pitty.

Is there any update on this? I'm using ArangoDB, but this feature would make it drastically more beneficial. I'm also in the process of trying to sell other developers on using ArangoDB, but they are leaning toward MongoDB simply because it has this feature.

As a work around, i am using WAL. It works well for me as i wait for the inbuilt event triggers in ArangoDB.

@gaithoben that's what I had been using before when my Scarango driver was just using HTTP calls to the API, but now that I'm wrapping the Java driver, it doesn't appear quite that simple, and polling for changes is a pain in the neck vs receiving events via subscription.

I am using the Js driver.  Lets hope it is fixed soon.

Don't worry. They have no plans to fix this ever. They have "better" things to worry about than a popular issue from free users..

😂😂 Not sure if the feature is available for enterprise users either. I think its quite something in terms of implementation. Either way so far, ArangoDb is the best database i have ever known by far!

@gcameo Yes, this is an old request and I understand that people are frustrated that this is still not possible. I don't know how the priority of this ticket has been handled in the past (I only joined a bit more than a year ago). but we do have it on our roadmap now. However, for a cluster setup this is highly non-trivial. We are currently working on a new replication protocol that will also be the building block for this feature. This will take a bit more time (no ETA), but we will get there eventually. :)

When I started following this ticket, I was working on a DARPA project. They'd added this functionality to their home-grown database, and performance was terrible. But this feature was absolutely vital, and its lack meant that I could never propose ArrangoDB as an alternative.
I don't have any idea where that project has gone since I left.
I'm excited to hear that something is finally happening with it.

So far I have been following this issue for a long time but it seems only a few updates. I then switch the mindset and start asking why we would need the real-time built-in DB. Like previous statements, the distributed capability is a higher priority of ArangoDB. It's also hard to scale real-time features within DB, it can scale better if we separate the real-time feature by applying a real-time message queue (such as NATS or Kafka), leaving ArangoDB as just storage to query some special queries. If we need WAL/CDC, it's also not scalable, because (my limited understanding) afaik the WAL/CDC is likely for syncing to different services/storage/DB. If that's the usage you're applying, maybe applying the stream processing before putting data to ArangoDB will be easier.
Even though the marketing point of ArangoDB is now targeting "storage and pipeline for ML", it's still acting as the only data store and graph query engine for future specific queries. For an enterprise context, we won't rely on single storage/DB for all queries. So if we use ArangoDB as the first storage for everything, it may not work very well.
For the use cases of live queries, this is a dedicated feature we would love to have. Some graph queries can take a long time, however, when working on real requirements, it turns out we can limit the depth of queries or some optimization ways to match business requirements, sometimes we need to cold-start by pre-calculating (such as recommendation results), without live-queries, benchmarking and applying analyzer to have estimated time should help (and that's what we should implement as normal developer task anyway). So by making clear requirements with time constraints, we may not need to poll for the result.
With such thoughts, I think applying ArangoDB is a matter of architecture perspective, not a real show-stopper without realtime feature. In addition to that, unless you are working on a small project, for future scale, please re-consider the use of Foxx as well.

@xgenvn Let's say that I have a query about all the paths from node foo to node bar. This gives me subgraph baz.
Now someone else adds edge quux which creates a new path from foo to bar. This should add a new subgraph to my query results.
Getting notified about the event over kafka doesn't help, unless you also have the entire graph cached locally. And then you still need to have an efficient graph query engine to realize it.
Maybe this is above and beyond what ArrangoDB should do. The fact that people still consider this feature request a blocker for adoption after nearly 9 years suggests that maybe the work-arounds are not as easy as you think.

Has anyone implemented anything like this and tried it out?


Error when trying to run 03_fwi.ipynb with GPU

Dear Devito team,
I am trying to run the tutorial notebook /devito/examples/seismic/tutorials/03_fwi.ipynb on my machine. It runs normally when using the CPU, but I would like to use the GPU. I tried setting %env DEVITO_PLATFORM=nvidiaX, but I encountered an error.
/tmp/devito-jitcache-uid1009/d3a3d2e6153c3468b5bd394c5b3e8c0ed9ec93cb.c: In function ‘initdamp’:
/tmp/devito-jitcache-uid1009/d3a3d2e6153c3468b5bd394c5b3e8c0ed9ec93cb.c:152:10: warning: ‘’ is used uninitialized in this function [-Wuninitialized]
152 |   return 0;
|          ^
/tmp/devito-jitcache-uid1009/d3a3d2e6153c3468b5bd394c5b3e8c0ed9ec93cb.c:152:10: warning: ‘’ is used uninitialized in this function [-Wuninitialized]
/tmp/devito-jitcache-uid1009/d3a3d2e6153c3468b5bd394c5b3e8c0ed9ec93cb.c:152:10: warning: ‘’ is used uninitialized in this function [-Wuninitialized]
/tmp/devito-jitcache-uid1009/d3a3d2e6153c3468b5bd394c5b3e8c0ed9ec93cb.c:152:10: warning: ‘’ is used uninitialized in this function [-Wuninitialized]
/tmp/devito-jitcache-uid1009/d3a3d2e6153c3468b5bd394c5b3e8c0ed9ec93cb.c:152:10: warning: ‘’ is used uninitialized in this function [-Wuninitialized]
/tmp/devito-jitcache-uid1009/d3a3d2e6153c3468b5bd394c5b3e8c0ed9ec93cb.c:152:10: warning: ‘’ is used uninitialized in this function [-Wuninitialized]
/tmp/devito-jitcache-uid1009/d3a3d2e6153c3468b5bd394c5b3e8c0ed9ec93cb.c:152:10: warning: ‘’ is used uninitialized in this function [-Wuninitialized]
/tmp/devito-jitcache-uid1009/d3a3d2e6153c3468b5bd394c5b3e8c0ed9ec93cb.c:152:10: warning: ‘’ is used uninitialized in this function [-Wuninitialized]
Output exceeds the size limit. Open the full output data in a text editor
CompileError                              Traceback (most recent call last)
Cell In[3], line 9
6 spacing = (10., 10.)  # Grid spacing in m. The domain size is now 1km by 1km
7 origin = (0., 0.)  # Need origin to define relative source and receiver locations
----> 9 model = demo_model('circle-isotropic', vp_circle=3.0, vp_background=2.5,
10                     origin=origin, shape=shape, spacing=spacing, nbl=40)
12 model0 = demo_model('circle-isotropic', vp_circle=2.5, vp_background=2.5,
13                      origin=origin, shape=shape, spacing=spacing, nbl=40,
14                      grid = model.grid)
16 plot_velocity(model)
File ~/anaconda3/envs/demo/lib/python3.9/site-packages/examples/seismic/preset_models.py:247, in demo_model(preset, **kwargs)
244     y, x = np.ogrid[-a:shape[0]-a, -b:shape[1]-b]
245     v[xx + yy <= r*r] = vp
--> 247     return SeismicModel(space_order=space_order, vp=v, origin=origin, shape=shape,
248                         dtype=dtype, spacing=spacing, nbl=nbl, bcs="damp",
249                         fs=fs, **kwargs)
251 elif preset.lower() in ['marmousi-isotropic', 'marmousi2d-isotropic']:
252     shape = (1601, 401)
File ~/anaconda3/envs/demo/lib/python3.9/site-packages/examples/seismic/model.py:273, in SeismicModel.init(self, origin, spacing, shape, space_order, vp, nbl, fs, dtype, subdomains, bcs, grid, **kwargs)
271 def init(self, origin, spacing, shape, space_order, vp, nbl=20, fs=False,
272              dtype=np.float32, subdomains=(), bcs="mask", grid=None, **kwargs):
--> 273     super(SeismicModel, self).init(origin, spacing, shape, space_order, nbl,
...
208 print("FAILED compiler invocation:" + " ".join(cc_cmdline),
209       file=sys.stderr)
--> 210 raise CompileError("module compilation failed")
CompileError: module compilation failed
I have read some issues and set %env DEVITO_ARCH=gcc but still filled.
Thank you for your help.
Best regards,
yym

To run on nvidia gpus with DEVITO_PLATFORM=nvidiaX you need to set the arch and language to valid ones, namely DEVITO_LANGUAGE=openmp with DEVITO_ARCH=nvc or DEVITO_LANGUAGE=openacc with DEVITO_ARCH=nvc
You can also use clang instead of nvc if you built llvm yourself  to support openmp/openacc offloading

Thank you for your help! I’m new to NVIDIA and Devito. I have added %env DEVITO_PLATFORM=nvidiaX  %env DEVITO_LANGUAGE=openacc  %env DEVITO_ARCH = nvc and got the nvc++ compiler isn't available on this system  Do I need to install the NVIDIA HPC SDK to use GPU on DeVito?

Yes you will need either  the SDK or another way to get the Nvidia compiler suite

Maybe this can be closed? Any updates?

Ι think this can probably be safely closed; feel free to reopen it if the issue remains or is still relevant.


Implement embeddings for use with LLM agents

Uses duckdb instead of chromadb. Currently subsets embeddings by table name. Not sure how to handle ephemeral tables yet.

import io
import asyncio

import lumen.ai as lmai
import panel as pn
import pandas as pd

from lumen.sources.duckdb import DuckDBSource
from lumen.ai.analysis import Join

pn.extension(
    "tabulator",
    "codeeditor",
    "vega",
    inline=False,
    template="fast",
    notifications=True,
)

llm = lmai.llm.OpenAI(
    # interceptor_path="interceptor.db"
)

embeddings = lmai.embeddings.OpenAIEmbeddings.from_dict(
    {
        "windturbines.parquet": [
            "The creator of this dataset is named Andrew HH",
            "To run the imaginary, test analysis, simply divide turbine count by 3",
        ]
    }
)

cs = lmai.memory["current_source"] = DuckDBSource(
    tables=["windturbines.parquet"],
    uri=":memory:",
    initializers=["INSTALL httpfs;", "LOAD httpfs;"],
)

custom_agent = lmai.agents.AnalysisAgent(analyses=[Join])

assistant = lmai.Assistant(
    llm=llm,
    agents=[
        lmai.agents.SourceAgent,
        lmai.agents.TableAgent,
        lmai.agents.TableListAgent,
        lmai.agents.SQLAgent,
        lmai.agents.PipelineAgent,
        lmai.agents.hvPlotAgent,
        lmai.agents.ChatAgent,
        custom_agent,
    ],
    embeddings=embeddings,
    suggestions=["Who is the creator of this dataset?"],
)
assistant.servable("Lumen.ai")
assistant.controls().servable(area="sidebar")

Superseded by https://github.com/holoviz/lumen/pull/764


Dump zuul vars and inventory in molecule jobs

As a pull request owner and reviewers, I checked that:

[x] Appropriate testing is done and actually running
[x] Appropriate documentation exists and/or is up-to-date

/approve
/lgtm
will force-merge. Molecule error isn't related to the change, nor the deployment one (the change doesn't affect deployment jobs).


Linear regression example

Hi, Many thanks for the great work. I was wondering if a simple linear regression example could be added to the tutorial/examples directory. I'm happy to do a PR.
Code

import autograd.numpy as np
import matplotlib.pyplot as plt

n_samples = 50
X_data = np.linspace(1, 50, n_samples).reshape((n_samples, 1))
Y = 10*train_X + 6 +10*np.random.randn(n_samples)
Y = Y.reshape((n_samples, 1))
X = np.hstack([X_data, np.ones((n_samples, 1))])

def cost(w):
    pred = np.dot(X, w)
    return np.sqrt(((pred - Y) ** 2).mean(axis=None))

from autograd import grad
grad_cost= grad(cost)

W = np.random.randn()
b =np.random.randn()
Weights = np.array([W, b]).reshape((2, 1))
for i in range(50):
    print "*"*20
    print i
    print "*"*20
    print cost(Weights)
    Weights = Weights - grad_cost(Weights)*0.01
    print cost(Weights), Weights

@nipunbatra train_X is not defined...example is not correct if X_train = X_data...

Hi, I have written similar code in notebook, including some animation to show how the linear regression work. Wonder if it is suitable to put in the example directory. Here is the gist for the notebook, although you can't see the animation there.
https://gist.github.com/catethos/5aa8e5de02131c7ef7b40c1526c22a05


Add sleep(1) to wait until previous tx sent

What is this pull request?
Fix TxPipelineIntegrationTestFixture::sendTransactions()
Why do you implement it? Why do we need this pull request?
To solve the issue of sending multiple transactions in the test.
Being lack this fix, the test of https://github.com/hyperledger/iroha/pull/692 sometimes fails because of sending multiple transactions.
How to use the features provided in the pull request?
sendTransactions({tx1, tx2, tx3});

Details/Features

Add sleep(1) after irohad->getCommandService()->ToriiAsync(pb_tx, response);
Remove useless debug.

P.S.

To confirm the previous transaction sent, using StatusAsync() is better. But currently, for the reason that there is no way to know the details of transaction statuses, I decided to use sleep(1)

Codecov Report

Merging #709 into develop will increase coverage by <.01%.
The diff coverage is 0%.

@@             Coverage Diff             @@
##           develop     #709      +/-   ##
===========================================
+ Coverage    42.95%   42.96%   +<.01%
===========================================
  Files          307      307
  Lines        12496    12493       -3
  Branches      6936     6934       -2
===========================================
- Hits          5368     5367       -1
  Misses         707      707
+ Partials      6421     6419       -2

Impacted Files
Coverage Δ

...tion/pipeline/transfer_asset_inter_domain_test.cpp
41.66% <ø> (+1.28%)
:arrow_up:

.../pipeline/tx_pipeline_integration_test_fixture.hpp
40.62% <0%> (-0.65%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 839b2b0...b0d715c. Read the comment docs.

It's fine that txs goes without order here. Imo it's better sort sent proposal and compare with the base one

I think that's one of the options, on the other hand, this test is the integration test. So maybe we have to send transactions the same way as clients. i.e. we need to send transactions via Torii endpoint.
And, if sending transactions like:
sendTransactions({tx1, tx2});

tx2 depends on tx1. If tx1 is not processed, tx2 may be failed at stateful validation.
Also, we cannot send transactions like:
sendTransactions({tx1});
sendTransactions({tx2});

Because validate() functions need to be called only once in one test. So,
sendTransactions({tx1, tx2});
validate();

is ok, but
sendTransactions({tx1});
validate();
sendTransactions({tx2});
validate();

causes the test to fail.

The problem with the sleep is it doesn't solve the problem. Only increases chances of green test. This probably should be done either via sorting as I mentioned earlier, or via conditional variable.
And about your comment: maybe you're right, but it seems quite strange for me to predict order of asynchronous service. And dependent commands should be sent in one txs, so there's no "dependent txs"
That are my though, I also would like to hear any others from our team

That are my though, I also would like to hear any others from our team

Can anybody help?
sendTransactions() is created by @lebdron so could you help if possible? (ping)

That method is synchronous here, since Torii itself (gGRPC service) is not created, and the gRPC endpoint is called directly. What is asynchronous here - OrderingGateTransportGrpc::propagate_transaction method:
https://github.com/hyperledger/iroha/blob/15ffc458b9704e19353ddba58b3582cf2202eaa9/irohad/ordering/impl/ordering_gate_transport_grpc.cpp#L53-L58
Therefore, I would suggest to inherit from ordering service/ordering service transport, add condition variable to inherited class, and notify when onTransaction method is called to ensure transaction order.

Sorry, my todo comment: Replace this with StatusAsync() after we can use tx status ON_PROCESS. is wrong.
If clients want to guarantee the order of transactions, they need to wait for COMMITTED from StatusAsync(). And we can use this method.
Is this not related to your comment? @lebdron

This PR stopped because we may implement https://github.com/hyperledger/iroha/pull/709#issuecomment-346259310 modifying ordering service init code. This design is being discussed now.

The scenario of sending transactions is no problem.
So I think:

Bad: Usesleep() to wait transactions are committed.
Not so good: Use StatusAsync() because it can guarantee transactions are committed.
Better: Change OrderingService (maybe OrderingInit?) implementation because polling is not used in the test.

I think it's better to use the way of 2 temporarily because I can quickly fix the problem.
And add TODO comment to replace it with the way of 3.

In order to wait transactions are committed,
the way of sleep(1) is bad. And StatusAsync() requires polling so there are same problem with sleep(1)
The scenario is no problem but removing the test is better?

@lebdron
Custom implementation for OrderingService in Txpipeline test requires duplicate code and maybe it is difficult to manage. The custom implementation will be such like:
// tx_pipeline_integration_test_fixture.hpp
class CustomOrderingServiceTransportGrpc final : public iroha::ordering::OrderingServiceTransportGrpc {
 public:
  grpc::Status OrderingServiceTransportGrpc::onTransaction(
      ......) {
    if (subscriber_.expired()) {
      log_->error("No subscriber");
    } else {
      subscriber_.lock()->onTransaction(*factory_.deserialize(*request));
      {
        std::lock_guard<std::mutex> lk(send_tx_mtx);
        send_tx_ready = true;
      }
      send_tx_cv.notify_one();
    }
    ......
  }
};

class CustomOrderingInit final : public iroha::network::OrderingInit {
 public:
  std::shared_ptr<ordering::OrderingGateImpl> initOrderingGate(
      const std::shared_ptr<ametsuchi::PeerQuery> &wsv,
      size_t max_size,
      std::chrono::milliseconds delay_milliseconds) override {
    ...
    ordering_service_transport = std::make_shared<ordering::CustomOrderingServiceTransportGrpc>();
    ...
  }
};

How to solve this problem?
Also, sleep() implementations is here, maybe better than polling:
https://github.com/hyperledger/iroha/blob/19651930fa6851586bfb9f09c3f6f2f3d3a51eaf/test/module/irohad/ordering/ordering_gate_service_test.cpp#L95

@lebdron @l4l
To sort transactions, I tried to implement CustomOrderingServiceTransportGrpc for test, but finally, I thought using std::this_thread::sleep_for is the best way.
The reasons are:

Sorting transactions requires change OrderingInit to an abstract class, which needs duplicate implementations only for initialization of ordering service classes. Using template class also changing codes tricky.
If we implement CustomOrderingServiceTransportGrpc inherited from OrderingServiceTransportGrpc, hiding onTransaction(), changing OrderingServiceTransportGrpc to an abstract class and duplicate implementations. Also, I think it's not a good way that changing OrderingServiceTransportGrpc functions to virtual ones like application.hpp, because of https://github.com/google/googletest/blob/master/googlemock/docs/CookBook.md#alternative-to-mocking-concrete-classes

I found there is a similar case of this PR. It uses this_thread::sleep_for to confirm order. So changing this code is also needed if we need to implement sorting transactions in OrderingServiceTransport, which takes some cost and it's not so much effective.
https://github.com/hyperledger/iroha/blob/develop/test/module/irohad/ordering/ordering_gate_service_test.cpp#L94-L95

I changed the code to use condition variable for waiting transactions.
When a transaction is committed, cv.notify_all() works via PeerCommunicationService

Can anyone else review this?


issue installing with conda

Hi!
When using conda install -c acoular acoular I get 3 subsequent SafetyErrors:

SafetyError: The package for acoular located at ... appears to be corrupted. The path X has a sha256 mismatch.

with X:

'Scripts/CalibHelper-script.py'
'Scripts/ResultExplorer-script.py'
'Scripts/acoular_demo-script.py'

followed by 4 ClobberErrors

ClobberError: This transaction has incompatible packages due to a shared path.
packages: conda-forge::jupyter_core-4.4.0-py_0, conda-forge::jupyter-1.0.0-py_1
path: 'lib/site-packages/jupyter.py'

ClobberError: This transaction has incompatible packages due to a shared path.
packages: conda-forge::jupyter_core-4.4.0-py_0, conda-forge::jupyter-1.0.0-py_1
path: 'lib/site-packages/pycache/jupyter.cpython-36.pyc'

ClobberError: This transaction has incompatible packages due to a shared path.
packages: conda-forge::ipykernel-5.1.0-py36h39e3cac_1001, conda-forge::widgetsnbextension-3.4.2-py36_1000
path: 'scripts/jupyter-kernel-script.py'

ClobberError: This transaction has incompatible packages due to a shared path.
packages: conda-forge::ipykernel-5.1.0-py36h39e3cac_1001, conda-forge::widgetsnbextension-3.4.2-py36_1000
path: 'scripts/jupyter-kernel.exe'

I've tried removing the cached package, and also just installing on another system that didn't have acoular from before. This didn't help, so I wonder if something is wrong with the latest Acoular library on conda.

This problem seems to appear recently when you try installing Acoular on Windows systems and will be fixed in the next release. Have you checked whether Acoular has not actually been installed nonetheless? (e.g., type "import acoular" in a python console)

Great to hear a fix is on its way!
Yes, I forgot to say, afterwardsthe install works fine, but installing it this way takes forever.


Give exception back to wasm

As of now, our shim wrapper functions for wasm does not give exception back. We are mostly ignoring exceptions or just returning -1 as function result.
For example __get_state wrapper for wasm: https://github.com/kleash/wasmer-chaincode-test/blob/bfc746d12cd70b67034fd9a5e83d37d26d14b0af/wasmcc/main.go#L98
Here we just ignore the error and returns -1 if we are unable to retrieve from the state.
A mechanism is required to propagate this exception back to wasm so that wasm chaincode developers can use the exception message and stack trace for debugging.

https://hacks.mozilla.org/2019/11/multi-value-all-the-wasm/
multi values might help with this and other things in the future. Maybe information to save for the future.

Yeah, this would be great.
But the blog points out that only Cranelift and Wasmtime supports it. Life doesn't support it as of now..

Just randomly seen this. We will soon add support for multi-values in Wasmer :)

@syrusakbary That seems interesting. We tried wasmer in our initial testing but it was not compatible with fabric sdk out of the box.

We tried wasmer in our initial testing but it was not compatible with fabric sdk out of the box

Wasmer supports Emscripten and WASI out of the box, but it should be very easy to plug new ABIs/SDKs. Let me know if we can help in any way, I'd be super happy to help!


SVGDOM implementation (uncomplete)

jsdom does not implement SVG elements atm. I would like to change that and thats why I try to kickoff the implementation here.
Its my first day contributing to jsdom so I am sure I did some mistakes. Please correct me whenever you spot something odd.
I will add commits to this PR whenever I finished an element.
The first commit only adds the SVGElement as basic implementation.
This PR is related to #1423

Thanks, this is very much appreciated! I'll try to review and see why Travis is failing.

Travis is failing because some lint errors.
This will also need some tests before it can be merged. See https://github.com/tmpvar/jsdom/blob/master/Contributing.md#web-platform-feature-tests

Yes, I saw the errors. I forgot a semicolon somewhere.
I will add tests as soon as I understand how that works.
Thanks for reviewing!

I could need a bit help with these tests.
I dont know where to start and especially where to place my tests.

You can place them in a new subdirectory of https://github.com/tmpvar/jsdom/tree/master/test/web-platform-tests/to-upstream called "svg".

How can I test objects which have no constructor?
In this case I want to write tests for the interface SVGStringList.
The normal conctructor cant be used (illegal constructor).  So how to do that?

Well, are there any APIs that return a SVGStringList on the platform? If so, use them. If not, then there's no reason to add SVGStringList :).

Well yes. SVGTests returns a stringlist (SVGGraphics implements SVGTests). But I feel uncomfortable using such a long way to get to the strings instead of just testing them straight forward

Nah, it's fine :)

This is from the specs about requiredExtensions:

If the attribute is not present, then it implicitly evaluates to "true". If a null string or empty string value is given to attribute ‘requiredExtensions’, the attribute evaluates to "false".

https://svgwg.org/svg2-draft/struct.html#RequiredExtensionsAttribute
I checked. In FF an empty Stringlist is returned. No false and no true. I am not sure which is the correct behavior.
PS: Sorry for bothering you with that :D

No problem! I think the confusion there is that it is talking about the requiredExtensions content attribute, i.e. the thing in the SVG markup itself: requiredExtensions="". The requiredExtensions IDL attribute is defined here: https://svgwg.org/svg2-draft/types.html#__svg__SVGTests__requiredExtensions
This gets kind of complicated, and needs the rules in https://svgwg.org/svg2-draft/types.html#TermReflect :-/.

The "evaluates to true/false" bit is presumably about how other parts in the spec will say something like "if the requiredExtensions attribute for this evaluates to true", which is a spec-ese way of saying "if all the required extensions are supported." I think.

I read through this and I think I understand it. However one question remains: how does the Object knows which attribute it is reflecting. Same goes for readonly. How does e.g. a list knows if it should be readonly?

How does e.g. a list knows if it should be readonly?

It doesn't need to. readonly doesn't mean you can't modify the list, it means you can't modify the reference to the list as in:
obj.list = new List(); // illegal
obj.list.clear(); // totally fine

The webidl2js layer ensures that readonly are enforced automatically, so you won't have to explicitly pay attention to that.

That makes totally sense. Thanks.
Left is the question about the reflectable List.

Do you mean how we know which properties are also reflected as attributes?
That's sadly not in the webidl language, but rather hidden in the spec language, so actual reading required :confused:
Chromium/WebKit has their own proprietary attribute extension which introduces [Reflect] as an extra argument into webidl as you can see here. This argument is also supported by webidl2js, so with some creative merging of the spec IDL, spec language and chromium idl, this is also taken care of mostly.

no. I know that the specs lay out which attributes have to be reflected.
I will try an exampel:
The attribute requiredExtensions has to be reflected. It is a StringList so adding one item to the list whould also add one Extension to the requiredExtensions attribute and the other way round.
My question is where i implement this behavior. The List is aware of beeing edited (insertItemBefore, removeItem... called) but it has no knowledge about which attribute it is representing. So where is the logic placed for actual updating the dom?

Hi folks!  I'm very interested in getting this patch into jsdom, so that I can use svg.js to build SVG in Node (see https://github.com/wout/svg.js/issues/464 as well as this article).  Anything I can do to help?  More test writing or are there other issues?

hey edemaine, I actually felt a bit lost in this huge specifications with all this reflection stuff. However - feel free to add your knowledge here. I would be really happy about that!

I think the main things left to do for landing this initial patch are:

Make sure it conforms to the SVG2 spec. (See above discussions about attribute reflection and things that have been removed above.)
Add to-upstream tests for every new piece of code, in web platform tests format. See Contributing.md for more details.

Feel free also to implement a subset, e.g. a basic SVGSVGElement class in a single patch, instead of @Fuzzyma's SVGTests and SVGStringList. That might be an easier patch to land.

OK, I finally read through this discussion.  Fuzzyma's main question seems to be how to make something like elt.requiredExtensions.appendItem('extension') tell elt to update its requiredExtensions.  I don't quite understand the issue in this case, because elt.requiredExtensions presumably is just a pointer to a single SVGStringList object, so changing that object will affect elt (and it's not like we're actually rendering SVG so there's no UI to update in this case).  So my sense is that reflection is "free" as far as dom.js is concerned.  (If this were an issue, we could subclass SVGStringList to store the element it belongs to and pass updates along... but this doesn't seem necessary.)  Am I missing something?
Also, I'm a little new to Github -- is the proper way to extend a pull request to clone the new repository, make my own commits, and then submit another pull request?

Also, I'm a little new to Github -- is the proper way to extend a pull request to clone the new repository, make my own commits, and then submit another pull request?

Yeah, that sounds good.

Hello! I'm currently working on a project that I'd prefer to use jsdom over phantomjs for. What's the status on this PR/what could I do to help move it along?

Sorry, I haven't had time to look more / work on this yet.  But looks like @Fuzzyma has had some time to write additional tests.  Perhaps he could report on what still remains?

@edemaine no worries. It looks like @nhunzaker is also working on this now. Maybe we could all figure out an MVP for this PR since the W3 spec for SVGs is huge and divide up the work? I'm particularly interesting in seeing #getBBox get implemented which relies on SVGRect, but some of the other classes that were started in here would be nice as well.
CC @Fuzzyma @domenic @Sebmaster

Digging more into this, I think figuring out a baseline to unblock all of our specific desires is a great starting point.
I simply want SVGElement to be defined on the global namespace with basic DOM operations. However I'd be happy to help collaborate on other pieces.

As far as I can remember my code, the "SVGElement is defined" goal was already achieved^^

@nhunzaker Agreed, it'd be nice to release this in pieces, if possible. Personally, I just want enough for svg.js to run, which I think is very little. Then we can break up the remaining work by sections of the SVG DOM spec.

@Fuzzyma can you mark this is fixing #2001 ?

Might fix #2001

I'm working on creating a cleaned-up version of the SVG-support PRs. Hopefully will finish today!


Instagram: Move the Get Stories Function to Background Task

Issue: https://github.com/animemoeus/backend/issues/480

Local Testing Result

Adding a field to enable/disable auto-download stories for each user

Adding background task function to periodic tasks

instagram.tasks.get_instagram_users_stories

Celery result

Saved story result

Post Deployment Testing
Auto Download Story Field ✅

Periodic Task ✅

Celery ✅

Stories ✅

PASS ✅ 👍


In openpai, how to download containers in harbor when submitting tasks

Organization Name:haiyangdaxue

Short summary about the issue/question:How to set the harbor private warehouse when submitting tasks in openpai so that tasks can run in custom containers
Brief what process you are following:

How to reproduce it:

OpenPAI Environment:

OpenPAI version:
Cloud provider or hardware configuration:
OS (e.g. from /etc/os-release):
Kernel (e.g. uname -a):
Hardware (e.g. core number, memory size, storage size, GPU type etc.):
Others:

Anything else we need to know:

just deploy a private docker registry. https://docs.docker.com/registry/deploying/

just deploy a private docker registry. https://docs.docker.com/registry/deploying/

Closing the issue as the question has been answered, feel free to reopen if you (@mtzfei) have further questions, thanks.


Add missing p-codes for admin1 and admin2 in OP and POP views.

The views for 3W and population were missing P-codes for admin1 and admin2. This correction adds them in.
Note that we'll have to synchronise this change with SQLAlchemy.

Added some more missing fields.