Build and deploy python library cpgclientlib

Travis currently only builds and deploys the scala code in this repository. cpgclientlib needs to be built and deployed to PyPi. Alternatively, it might be a better choice to simply host that library in another repository.

Python tests can currently be run as follows:
cd cpgclientlib
./runtests.sh

The directory cpgclientlib contains a python wheel with a setup.py. setup.py determines the version of codepropertygraph from git automatically.
@mpollmeier do you think it's a good idea if we publish artifacts automatically to PyPi, or should we maybe offer them via github releases? Opinions?

if cpgclientlib naturally belongs into this repo and is released together with codepropertygraph, it sounds like we should build and release it together. here's the docs for travis-ci and python (see .travis.yml in the repo): https://docs.travis-ci.com/user/languages/python/
That being said, I've never deployed anything to PyPi, so I don't know what's involved.
Suggestion: have a look at that docs page and see if you can configure the .travis.yml to do what you want. If it's unclear, can you list the steps necessary in order to publish to PyPi? Do we need some credentials and gpg keys for that?

also note https://docs.travis-ci.com/user/deployment/pypi/

Yes, cpgclientlib naturally belongs to codepropertygraph and the two should be released together. That's also why I believe using the same version number makes sense. I had actually reviewed the docs you had posted earlier, but I was wondering how we can Jenkins to build, test, and release both the Scala and the Python code. It seems like we can only specify one language. I did see that there is the possibility to specify a build matrix to possibly resolve this.  Also, do you have a local travis setup with which you test travis changes? I fear testing out travis builds directly on master and breaking everything for several hours.

do you have a local travis setup with which you test travis changes? I fear testing out travis builds directly on master and breaking everything for several hours.

Just add your branch to the whitelist of branches here: https://github.com/ShiftLeftSecurity/codepropertygraph/blob/master/.travis.yml#L6
It will only run the test stage, not release, because of this condition: https://github.com/ShiftLeftSecurity/codepropertygraph/blob/master/.travis.yml#L31-L32
There's a few ways on how to handle multiple languages, e.g.

trigger the python release from sbt, i.e. https://github.com/ShiftLeftSecurity/codepropertygraph/blob/master/.travis.yml#L39 would include another sbt task like releaseCpgClientLib, which you add to the build and e.g. call a shell file or so
figure out how to run multiple commands in one script and make it part of the release script in https://github.com/ShiftLeftSecurity/codepropertygraph/blob/master/.travis.yml#L39
add another stage releaseCpgClientLib that just calls your script

Sounds like the last option is best - give it a try, it should automagically start building your branch when you push it (and you added it to the whitelist).

yep, I'm experimenting with just calling python-related testing and release in this PR here: https://github.com/ShiftLeftSecurity/codepropertygraph/pull/185 - still failing on travis, but I think I'll get there some time today.

Building/testing already works: https://github.com/ShiftLeftSecurity/codepropertygraph/pull/185

This is finally working.


"Edit" option should not be presented in "OpenLDAP" access control page.

Rancher versions:v1.6.6-rc3
Steps to reproduce the problem:
Enable OpenLDAP access control.
Navigate to "Admin/Access Control".
"Edit" option is presented in "OpenLDAP" access control page.
Expected Behavior:
"Edit" option should not be presented in "OpenLDAP" access control page.

Version - 1.6.6-rc4
Verified fixed


Use GSS code as SNAC for Belfast

This finishes the work in #918 and aligns the Belfast council with the rest of the NI councils to use GSS code in place of the SNAC.  This assumes we have switched over to use the new mapit database.
This includes the commits for #918, once that's merged we'll rebase.
For: https://trello.com/c/ilFPO330/90-switch-over-to-use-new-mapit-in-production
Background: https://trello.com/c/kjurG3i8/302-support-new-northern-ireland-local-authorities-for-licensing

Once this is merged and looks OK in integration, we can deploy soon after https://github.com/alphagov/govuk-puppet/pull/4220

:+1:  Merge when ready


Resolve warning about use of Base.Nothing in v0.4.0

using Gadfly works on v0.4.0 and v0.3.2
v0.4.0 stil gives the following
WARNING: eval from module Compose to Gadfly:
Expr(:import, :Patchwork)::Any
  ** incremental compilation may be broken for this module **

Just saw #725 took care of this already... Closing


Thanks for contribution, @breyed

Updated nuget package released and available on nuget.org


Support for RPi2

Please shut this down immediately if it's a non-starter, but what would it take to support RPi2 (Armv7)?
I managed to bumble my way to installing NixOS with a pre-made 19.03 image, but any attempt to rebuild (ie. put cachix or turn SSH on) fails horribly due to the low RAM capability on device. My attempts to build an image on my Mac haven't worked thus far (though I am deeply out of my depth), but many of the tools you're using are the same as I'm trying to understand.
Many thanks for your work!

Hi!
Thanks for considering using this! I would love to, but there are a few key obstacles:

I don't have a RPi 2 on hand! :p so obviously I can't really test if what I'm doing makes sense.
This repo is specifically tailored for the AArch64 architecture, which the Pi 2 doesn't support.
The default Nix caches don't have any package for armv7, but only amd64 and aarch64.

I won't say that this is impossible, but convoluted! So to do this you would need to try to:

In the QEMU emulation container, you'd need to replace the usage of qemu-aarch64-static with probably something similar to qemu-arm-static (see here for a list of executables).
https://github.com/Robertof/nixos-docker-sd-image-builder/blob/a53d1124f71a172f92767565f21100a0d78b4424/docker/setup-qemu/build/setup-image.sh#L34
https://github.com/Robertof/nixos-docker-sd-image-builder/blob/a53d1124f71a172f92767565f21100a0d78b4424/docker/setup-qemu/scripts/setup-qemu.sh#L5
2. Then, you'd need to update the magic masks used to interpret aarch64 executables with the ARM masks available here.
https://github.com/Robertof/nixos-docker-sd-image-builder/blob/a53d1124f71a172f92767565f21100a0d78b4424/docker/setup-qemu/scripts/update-binfmt.sh#L4-L7
Remove/replace the aarch64-tester with an ARMv7 binary or just get rid of this altogether (a sleep would suffice):
https://github.com/Robertof/nixos-docker-sd-image-builder/blob/a53d1124f71a172f92767565f21100a0d78b4424/docker/build-nixos/wait-for-qemu.sh#L1-L9
Configure any ARMv7 cache here, perhaps either this cache or this.
Finally, change the Nix build command line with an appropriate system argument and perhaps other changes I'm not aware of that might be needed:
https://github.com/Robertof/nixos-docker-sd-image-builder/blob/a53d1124f71a172f92767565f21100a0d78b4424/docker/docker-compose.yml#L39-L47

Obviously, these are high level guidelines and I'm not sure of what else can blow up. I'd love to experiment with this but I'm in a pretty busy period right now, so I have very little time to play around with that. If you have, though, please keep me posted -- I'm completely open at the idea of adapting this repo/creating another one to generate RPi 2 images as well!

Wow, thank you for your very considered response, especially since you're busy.
This does seem like a good place for me to start, I'm certainly keen to see how it pans out. I'll see how I get on, and will be sure to feedback successes (hopefully) I have!

Glad that my response was of inspiration! I'll do my best to help if anything comes up, thanks again for considering the use of this!

I've made a some changes, and things seem to be going in the right direction, but I'm stuck on this error:
build-nixos_1   | error: Package ‘nixos-sd-image-20.03post-git-arm-linux.img’ in /home/nixos/nixpkgs/nixos/modules/installer/cd-dvd/sd-image.nix:134 is not supported on ‘arm-linux’, refusing to evaluate.

Any idea what's happpening? I'd assume binfmt would have prevented it. Perhaps I've got the wrong architecture specified somewhere.
Also: what's stopping us from using these Rpi images as base-derivatives? I don't understand derivations enough...

I've made a some changes, and things seem to be going in the right direction, but I'm stuck on this error:
build-nixos_1   | error: Package ‘nixos-sd-image-20.03post-git-arm-linux.img’ in /home/nixos/nixpkgs/nixos/modules/installer/cd-dvd/sd-image.nix:134 is not supported on ‘arm-linux’, refusing to evaluate.

Any idea what's happpening? I'd assume binfmt would have prevented it. Perhaps I've got the wrong architecture specified somewhere.
Also: what's stopping us from using these Rpi images as base-derivatives? I don't understand derivations enough...

Hey! Huge apologies for the delay in getting back to you -- unfortunately things haven't gotten less busy recently!
That error is quite weird -- I suppose you're using the 20.03 branch which has really nothing much interesting there:
https://github.com/NixOS/nixpkgs/blob/release-20.03/nixos/modules/installer/cd-dvd/sd-image.nix#L134
The RPi image derivation is actually used as a base -- however, to be able to change some of the hardcoded options there I've just duplicated them in this tree!
What value did you use for --option system in docker-compose.yml? You should use --option system armv7l-linux. Regardless, I would suggest two course of actions here:

Try with the new value for 'system'. Also, I've just noticed that there is a sd-image-armv7l-multiplatform.nix derivation that you could copy to a new folder in config/ (updating the reference in config/sd-image.nix) and base yourself on!
if that doesn't work, you could try to scratch using this altogether and instead try to cross-compile. You'd need an existing installation of Nix (on x86/x86_64), on which you could then try to use this: https://github.com/illegalprime/nixos-on-arm#raspberry-pi-2 which looks very promising!

Unfortunately the documentation for building NixOS non-natively for ARMv7 devices is slowly shrinking as AArch64 rises, so documentation is quite sparse! Let me know if that works.
Thank you,
Roberto

Hey! Huge apologies for the delay in getting back to you -- unfortunately things haven't gotten less busy recently!
That error is quite weird -- I suppose you're using the 20.03 branch which has really nothing much interesting there:
https://github.com/NixOS/nixpkgs/blob/release-20.03/nixos/modules/installer/cd-dvd/sd-image.nix#L134
The RPi image derivation is actually used as a base -- however, to be able to change some of the hardcoded options there I've just duplicated them in this tree!
What value did you use for --option system in docker-compose.yml? You should use --option system armv7l-linux. Regardless, I would suggest two course of actions here:

Try with the new value for 'system'. Also, I've just noticed that there is a sd-image-armv7l-multiplatform.nix derivation that you could copy to a new folder in config/ (updating the reference in config/sd-image.nix) and base yourself on!
if that doesn't work, you could try to scratch using this altogether and instead try to cross-compile. You'd need an existing installation of Nix (on x86/x86_64), on which you could then try to use this: https://github.com/illegalprime/nixos-on-arm#raspberry-pi-2 which looks very promising!

Unfortunately the documentation for building NixOS non-natively for ARMv7 devices is slowly shrinking as AArch64 rises, so documentation is quite sparse! Let me know if that works.
Thank you,
Roberto


No syntax highlighting on .tmpl files

Gitea version (or commit ref): 1.16.0+dev-130-g0bd58d61e
Can you reproduce the bug at https://try.gitea.io:

[x] Yes:
[ ] No

Description
Gitea doesn't show any syntax highlighting on .tmpl files in the Web UI
See example: https://try.gitea.io/n/gitea/src/branch/main/templates/home.tmpl

The reason for this is that you can't guarantee the type of template from the file alone. I don't think there's a standard file extension for go templates, but if there were, then we could go based off of that.

I think it's actually supposed to be .go.tmpl
Realistically any heuristic measure is going to get things wrong from time to time so we should have a way to select the language and we should use git attributes for hints too.


From container image concept page, signpost readers to Kubernetes downloads

Revise Images concept in Kubernetes. This is a top match for the search term “Kubernetes container images”.
Help readers find Kubernetes container images for download [preview].

/retitle From container image concept page, signpost readers to Kubernetes downloads

/approve

/approve


feat: Expose gatekeeper OpenAPI spec #1502

Gatekeeper OpenAPI specification is accessible via Sandbox's GitHub Pages.
To navigate the spec:

Go to https://.github.io/sandbox (for this PR try https://ypukhta.github.io/sandbox/, after merge it would be https://trustbloc.github.io/sandbox)
Click on OpenAPI specifications > Gatekeeper

The support for GitHub Pages is enabled for each repo separately. An repo admin will need to enable it for the trustbloc/sandbox repo after the merge. See instructions here - https://tomcam.github.io/least-github-pages/enable-github-pages.html

@cla-bot check


[BUG] Script not catching all cards

MAKE SURE THE ERROR IS NOT FROM YOU, THIS IS THE MOST FREQUENT CASE
Error Log
Didnt get one because script "run as it should"
Screenshot of the page
No real error page. Rewards Dashboard just missing some green hooks after run the Script. They also dont work after a second time of running the script.
Source code of dashboard
https://gist.github.com/DerMuffin/07c66f80bfc54b13c92552d64bad5671

[DAILY SET] Trying to complete the Daily Set...
Traceback (most recent call last):
File "FolderPath\ms_rewards_farmer.py", line 798, in
completeDailySet(browser)
File "FolderPath\ms_rewards_farmer.py", line 487, in completeDailySet
d = getDashboardData(browser)['dailySetPromotions']
File "FolderPath\ms_rewards_farmer.py", line 483, in getDashboardData
dashboard = json.loads(dashboard)
File "FolderPath\AppData\Local\Programs\Python\Python39\lib\json_init_.py", line 346, in loads
return _default_decoder.decode(s)
File "FolderPath\AppData\Local\Programs\Python\Python39\lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "FolderPath\AppData\Local\Programs\Python\Python39\lib\json\decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

From what I can tell, it's the A, B, C quiz type of quests that are getting skipped, not running, or erroring out.

me is the card (More activities) and
(Quests and Loyalty Cards) What the Script Doesn't Do


Unable to index JSON from HDFS using SchemaRDD.saveToEs()

Elasticsearch-hadoop: elasticsearch-hadoop-2.1.0.BUILD-20150224.023403-309.zip
Spark: 1.2.0
I'm attempting to take a very basic JSON document on HDFS and index it in ES using SchemaRDD.saveToEs().  According to the documentation under "writing existing JSON to elasticsearch" it should be as easy as creating a SchemaRDD via SQLContext.jsonFile() and then index using .saveToEs(), but I'm getting an error.
Replicating the problem:

Create JSON file on hdfs with the content:
{"key":"value"}

Execute code in spark-shell
import org.apache.spark.SparkContext._
import org.elasticsearch.spark._
val sqlContext = new org.apache.spark.sql.SQLContext(sc)
import sqlContext._
val input = sqlContext.jsonFile("hdfs://nameservice1/user/mshirley/test.json")
input.saveToEs("mshirley_spark_test/test")

Error:
 org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest: Found unrecoverable error [Bad Request(400) - Invalid JSON fragment received[["value"]][MapperParsingException[failed to parse]; nested: ElasticsearchParseException[Failed to derive xcontent from (offset=13, length=9): [123, 34, 105, 110, 100, 101, 120, 34, 58, 123, 125, 125, 10, 91, 34, 118, 97, 108, 117, 101, 34, 93, 10]]; ]];

input object:
res1: org.apache.spark.sql.SchemaRDD =
SchemaRDD[6] at RDD at SchemaRDD.scala:108
== Query Plan ==
== Physical Plan ==
PhysicalRDD [key#0], MappedRDD[5] at map at JsonRDD.scala:47

input.printSchema():
root
 |-- key: string (nullable = true)

Expected result:
I expect to be able to read a file from HDFS that contains a JSON document per line and submit that data to ES for indexing.

Just FYI, using HDFS is not necessary to duplicate this issue.  Importing a local file has the same result.
val input = sqlContext.jsonFile("test.json")

@mshirley I cannot reproduce the problem. The problem likes with your input which seems to be read incorrectly.
Here's the test that I've tried on Spark 1.2 and 1.2.1:
val input = sqc.jsonFile(this.getClass.getResource("/simple.json").toURI().toString())
input.printSchema
println(input.schema)
input.saveToEs("spark-test/json-file")

Notice that the schema is much richer than yours:
root
 |-- address: struct (nullable = true)
 |    |-- streetAddress: string (nullable = true)
 |-- age: integer (nullable = true)
 |-- children: array (nullable = true)
 |    |-- element: string (containsNull = false)
 |-- firstName: string (nullable = true)
 |-- isAlive: boolean (nullable = true)

and
StructType(ArrayBuffer(StructField(address,StructType(ArrayBuffer(StructField(streetAddress,StringType,true))),true), StructField(age,IntegerType,true), StructField(children,ArrayType(StringType,false),true), StructField(firstName,StringType,true), StructField(isAlive,BooleanType,true)))

The data is saved(indexed) successfully to ES.
Maybe your data (field value) contains some special chars that are not escaped properly?

I can't duplicate your success using the same json document, so something else must be the issue.  Are you doing something different with the imports or with the sparkSQL context?
$ cat simple.json
{"address":{"streetAddress":"asdf"},"age":10, "children":["child1","child2","child3"], "firstname": "m", "isAlive": true}

This json is valid according to jsonlint.
$ spark-shell --jars ~/elasticsearch-hadoop-2.1.0.BUILD-SNAPSHOT/dist/elasticsearch-spark_2.10-2.1.0.BUILD-SNAPSHOT.jar
<snip>
scala> import org.apache.spark.SparkContext._
import org.apache.spark.SparkContext._

scala> import org.elasticsearch.spark._
import org.elasticsearch.spark._

scala>

scala> val sqlContext = new org.apache.spark.sql.SQLContext(sc)
sqlContext: org.apache.spark.sql.SQLContext = org.apache.spark.sql.SQLContext@6a6b602

scala> import sqlContext._
import sqlContext._

scala> val input = sqlContext.jsonFile(this.getClass.getResource("/simple.json").toURI().toString())
<snip>
input: org.apache.spark.sql.SchemaRDD =
SchemaRDD[6] at RDD at SchemaRDD.scala:108
== Query Plan ==
== Physical Plan ==
PhysicalRDD [address#0,age#1,children#2,firstname#3,isAlive#4], MappedRDD[5] at map at JsonRDD.scala:47

scala> input.printSchema
root
 |-- address: struct (nullable = true)
 |    |-- streetAddress: string (nullable = true)
 |-- age: integer (nullable = true)
 |-- children: array (nullable = true)
 |    |-- element: string (containsNull = false)
 |-- firstname: string (nullable = true)
 |-- isAlive: boolean (nullable = true)

scala> println(input.schema)
StructType(ArrayBuffer(StructField(address,StructType(ArrayBuffer(StructField(streetAddress,StringType,true))),true), StructField(age,IntegerType,true), StructField(children,ArrayType(StringType,false),true), StructField(firstname,StringType,true), StructField(isAlive,BooleanType,true)))

scala> input.saveToEs("spark-test/json-file")                                                                                [81/1985]
15/02/27 18:27:52 INFO BlockManager: Removing broadcast 1
15/02/27 18:27:52 INFO BlockManager: Removing block broadcast_1_piece0
15/02/27 18:27:52 INFO MemoryStore: Block broadcast_1_piece0 of size 1859 dropped from memory (free 277999146)
15/02/27 18:27:52 INFO BlockManagerInfo: Removed broadcast_1_piece0 on localhost:40090 in memory (size: 1859.0 B, free: 265.4 MB)
15/02/27 18:27:52 INFO BlockManagerMaster: Updated info of block broadcast_1_piece0
15/02/27 18:27:52 INFO BlockManager: Removing block broadcast_1
15/02/27 18:27:52 INFO MemoryStore: Block broadcast_1 of size 3176 dropped from memory (free 278002322)
15/02/27 18:27:52 INFO ContextCleaner: Cleaned broadcast 1
15/02/27 18:27:53 INFO SparkContext: Starting job: runJob at EsSpark.scala:51
15/02/27 18:27:53 INFO DAGScheduler: Got job 1 (runJob at EsSpark.scala:51) with 2 output partitions (allowLocal=false)
15/02/27 18:27:53 INFO DAGScheduler: Final stage: Stage 1(runJob at EsSpark.scala:51)
15/02/27 18:27:53 INFO DAGScheduler: Parents of final stage: List()
15/02/27 18:27:53 INFO DAGScheduler: Missing parents: List()
15/02/27 18:27:53 INFO DAGScheduler: Submitting Stage 1 (SchemaRDD[6] at RDD at SchemaRDD.scala:108
== Query Plan ==
== Physical Plan ==
PhysicalRDD [address#0,age#1,children#2,firstname#3,isAlive#4], MappedRDD[5] at map at JsonRDD.scala:47), which has no missing parents
15/02/27 18:27:53 INFO MemoryStore: ensureFreeSpace(6280) called with curMem=300234, maxMem=278302556
15/02/27 18:27:53 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 6.1 KB, free 265.1 MB)
15/02/27 18:27:53 INFO MemoryStore: ensureFreeSpace(3703) called with curMem=306514, maxMem=278302556
15/02/27 18:27:53 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 3.6 KB, free 265.1 MB)
15/02/27 18:27:53 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on localhost:40090 (size: 3.6 KB, free: 265.4 MB)
15/02/27 18:27:53 INFO BlockManagerMaster: Updated info of block broadcast_2_piece0
15/02/27 18:27:53 INFO SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:838
15/02/27 18:27:53 INFO DAGScheduler: Submitting 2 missing tasks from Stage 1 (SchemaRDD[6] at RDD at SchemaRDD.scala:108
== Query Plan ==
== Physical Plan ==
PhysicalRDD [address#0,age#1,children#2,firstname#3,isAlive#4], MappedRDD[5] at map at JsonRDD.scala:47)
15/02/27 18:27:53 INFO TaskSchedulerImpl: Adding task set 1.0 with 2 tasks
15/02/27 18:27:53 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 2, localhost, PROCESS_LOCAL, 1384 bytes)
15/02/27 18:27:53 INFO TaskSetManager: Starting task 1.0 in stage 1.0 (TID 3, localhost, PROCESS_LOCAL, 1384 bytes)
15/02/27 18:27:53 INFO Executor: Running task 0.0 in stage 1.0 (TID 2)
15/02/27 18:27:53 INFO Executor: Running task 1.0 in stage 1.0 (TID 3)
15/02/27 18:27:53 INFO HadoopRDD: Input split: file:/home/mshirley/simple.json:0+61
15/02/27 18:27:53 INFO HadoopRDD: Input split: file:/home/mshirley/simple.json:61+61
15/02/27 18:27:53 INFO Executor: Finished task 1.0 in stage 1.0 (TID 3). 1774 bytes result sent to driver
15/02/27 18:27:53 INFO TaskSetManager: Finished task 1.0 in stage 1.0 (TID 3) in 471 ms on localhost (1/2)
15/02/27 18:27:53 ERROR TaskContextImpl: Error in TaskCompletionListener
org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest: Found unrecoverable error [Bad Request(400) - Invalid JSON fragment received[[["asdf"],10,["child1","child2","child3"],"m",true]][
MapperParsingException[failed to parse]; nested: ElasticsearchParseException[Failed to derive xcontent from (offset=13, length=51): [123, 34, 105, 110, 100, 101, 120, 34, 58, 123, 125,
 125, 10, 91, 91, 34, 97, 115, 100, 102, 34, 93, 44, 49, 48, 44, 91, 34, 99, 104, 105, 108, 100, 49, 34, 44, 34, 99, 104, 105, 108, 100, 50, 34, 44, 34, 99, 104, 105, 108, 100, 51, 34,
 93, 44, 34, 109, 34, 44, 116, 114, 117, 101, 93, 10]]; ]]; Bailing out..
        at org.elasticsearch.hadoop.rest.RestClient.retryFailedEntries(RestClient.java:202)
        at org.elasticsearch.hadoop.rest.RestClient.bulk(RestClient.java:166)

        at org.elasticsearch.hadoop.rest.RestRepository.tryFlush(RestRepository.java:209)
        at org.elasticsearch.hadoop.rest.RestRepository.flush(RestRepository.java:233)
        at org.elasticsearch.hadoop.rest.RestRepository.close(RestRepository.java:246)
        at org.elasticsearch.hadoop.rest.RestService$PartitionWriter.close(RestService.java:129)
        at org.elasticsearch.spark.rdd.EsRDDWriter$$anonfun$write$1.apply$mcV$sp(EsRDDWriter.scala:40)
        at org.apache.spark.TaskContextImpl$$anon$2.onTaskCompletion(TaskContextImpl.scala:57)                                                                                [31/1985]
        at org.apache.spark.TaskContextImpl$$anonfun$markTaskCompleted$1.apply(TaskContextImpl.scala:68)
        at org.apache.spark.TaskContextImpl$$anonfun$markTaskCompleted$1.apply(TaskContextImpl.scala:66)
        at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
        at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
        at org.apache.spark.TaskContextImpl.markTaskCompleted(TaskContextImpl.scala:66)
        at org.apache.spark.scheduler.Task.run(Task.scala:58)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
15/02/27 18:27:53 ERROR Executor: Exception in task 0.0 in stage 1.0 (TID 2)
org.apache.spark.util.TaskCompletionListenerException: Found unrecoverable error [Bad Request(400) - Invalid JSON fragment received[[["asdf"],10,["child1","child2","child3"],"m",true]]
[MapperParsingException[failed to parse]; nested: ElasticsearchParseException[Failed to derive xcontent from (offset=13, length=51): [123, 34, 105, 110, 100, 101, 120, 34, 58, 123, 125
, 125, 10, 91, 91, 34, 97, 115, 100, 102, 34, 93, 44, 49, 48, 44, 91, 34, 99, 104, 105, 108, 100, 49, 34, 44, 34, 99, 104, 105, 108, 100, 50, 34, 44, 34, 99, 104, 105, 108, 100, 51, 34
, 93, 44, 34, 109, 34, 44, 116, 114, 117, 101, 93, 10]]; ]]; Bailing out..
        at org.apache.spark.TaskContextImpl.markTaskCompleted(TaskContextImpl.scala:76)
        at org.apache.spark.scheduler.Task.run(Task.scala:58)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
15/02/27 18:27:53 WARN TaskSetManager: Lost task 0.0 in stage 1.0 (TID 2, localhost): org.apache.spark.util.TaskCompletionListenerException: Found unrecoverable error [Bad Request(400)
 - Invalid JSON fragment received[[["asdf"],10,["child1","child2","child3"],"m",true]][MapperParsingException[failed to parse]; nested: ElasticsearchParseException[Failed to derive xco
ntent from (offset=13, length=51): [123, 34, 105, 110, 100, 101, 120, 34, 58, 123, 125, 125, 10, 91, 91, 34, 97, 115, 100, 102, 34, 93, 44, 49, 48, 44, 91, 34, 99, 104, 105, 108, 100,
49, 34, 44, 34, 99, 104, 105, 108, 100, 50, 34, 44, 34, 99, 104, 105, 108, 100, 51, 34, 93, 44, 34, 109, 34, 44, 116, 114, 117, 101, 93, 10]]; ]]; Bailing out..
        at org.apache.spark.TaskContextImpl.markTaskCompleted(TaskContextImpl.scala:76)
        at org.apache.spark.scheduler.Task.run(Task.scala:58)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

15/02/27 18:27:53 ERROR TaskSetManager: Task 0 in stage 1.0 failed 1 times; aborting job

15/02/27 18:27:53 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool
15/02/27 18:27:53 INFO TaskSchedulerImpl: Cancelling stage 1
15/02/27 18:27:53 INFO DAGScheduler: Job 1 failed: runJob at EsSpark.scala:51, took 0.627167 s
15/02/27 18:27:54 INFO BlockManager: Removing broadcast 2
15/02/27 18:27:54 INFO BlockManager: Removing block broadcast_2
15/02/27 18:27:54 INFO MemoryStore: Block broadcast_2 of size 6280 dropped from memory (free 277998619)
15/02/27 18:27:54 INFO BlockManager: Removing block broadcast_2_piece0
15/02/27 18:27:54 INFO MemoryStore: Block broadcast_2_piece0 of size 3703 dropped from memory (free 278002322)
15/02/27 18:27:54 INFO BlockManagerInfo: Removed broadcast_2_piece0 on localhost:40090 in memory (size: 3.6 KB, free: 265.4 MB)
15/02/27 18:27:54 INFO BlockManagerMaster: Updated info of block broadcast_2_piece0
15/02/27 18:27:54 INFO ContextCleaner: Cleaned broadcast 2
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1.0 failed 1 times, most recent failure: Lost task 0.0 in stage 1.0 (TID 2, localhost): org.apache.sp
ark.util.TaskCompletionListenerException: Found unrecoverable error [Bad Request(400) - Invalid JSON fragment received[[["asdf"],10,["child1","child2","child3"],"m",true]][MapperParsin
gException[failed to parse]; nested: ElasticsearchParseException[Failed to derive xcontent from (offset=13, length=51): [123, 34, 105, 110, 100, 101, 120, 34, 58, 123, 125, 125, 10, 91
, 91, 34, 97, 115, 100, 102, 34, 93, 44, 49, 48, 44, 91, 34, 99, 104, 105, 108, 100, 49, 34, 44, 34, 99, 104, 105, 108, 100, 50, 34, 44, 34, 99, 104, 105, 108, 100, 51, 34, 93, 44, 34,
 109, 34, 44, 116, 114, 117, 101, 93, 10]]; ]]; Bailing out..
        at org.apache.spark.TaskContextImpl.markTaskCompleted(TaskContextImpl.scala:76)
        at org.apache.spark.scheduler.Task.run(Task.scala:58)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

Driver stacktrace:
        at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1214)
        at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1203)
        at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1202)
        at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
        at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
        at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1202)
        at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:696)
        at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:696)
        at scala.Option.foreach(Option.scala:236)
        at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:696)
        at org.apache.spark.scheduler.DAGSchedulerEventProcessActor$$anonfun$receive$2.applyOrElse(DAGScheduler.scala:1420)
        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
        at akka.actor.ActorCell.invoke(ActorCell.scala:456)
        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)
        at akka.dispatch.Mailbox.run(Mailbox.scala:219)
        at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

Still can't reproduce it.
I've included the JSON you sent me and even add it in the trunk, see it here
Runs fine from both Eclipse and command-line:
12:59:44,320  INFO Executor task launch worker-0 rdd.HadoopRDD - Input split: file:/...basic.json:0+131
...
root
 |-- address: struct (nullable = true)
 |    |-- streetAddress: string (nullable = true)
 |-- age: integer (nullable = true)
 |-- children: array (nullable = true)
 |    |-- element: string (containsNull = false)
 |-- firstName: string (nullable = true)
 |-- isAlive: boolean (nullable = true)
...
StructType(ArrayBuffer(StructField(address,StructType(ArrayBuffer(StructField(streetAddress,StringType,true))),true), StructField(age,IntegerType,true), StructField(children,ArrayType(StringType,false),true), StructField(firstName,StringType,true), StructField(isAlive,BooleanType,true)))
...
PhysicalRDD [address#1,age#2,children#3,firstName#4,isAlive#5], MappedRDD[12] at map at JsonRDD.scala:47)
12:59:44,393  INFO Executor task launch worker-0 sql.EsSchemaRDDWriter - Writing to [spark-test/json-file]
12:59:44,432  INFO elasticsearch[Black Mamba][clusterService#updateTask][T#1] cluster.metadata - [Black Mamba] [spark-test] update_mapping [json-file] (dynamic)

gradle build integrationTest

completes just fine as does the CI
I'm not sure what else to tell you. I've tried this with different Elasticsearch versions (1.4.x) and everything works fine. Maybe your JSON encoding is incorrect or potentially you have some proxy in place. I realy don't know what else to tell you.
Try enabling logging on the rest package (see the logging section in the docs) and check out the information sent over the wire to Elasticsearch.
Cheers,

I am observing the same issue with Spark1.2.1 and parquet. Here is the code I'm running:
import org.apache.spark.sql.SQLContext
import sqlContext._

 sc.
            parallelize(1 to 1024).
            flatMap(i => Seq.fill(1024)(KeyValue(i, i.toString))).
            saveAsParquetFile("large.parquet")

            parquetFile("large.parquet").registerTempTable("large")
            sql("SELECT * FROM large").saveToEs("test/sparkes")

Elasticsearch version is 1.4.4
sbt file:
libraryDependencies ++= Seq(
        "org.apache.spark" %% "spark-sql" % "1.2.1" % "provided",
        "org.apache.spark" %% "spark-hive" % "1.2.1" % "provided",
        "org.apache.commons" % "commons-lang3" % "3.0",
        "org.elasticsearch" %% "elasticsearch-spark" % "2.1.0.BUILD-SNAPSHOT"
        )

Spark driver error:
15/03/23 12:26:04 INFO scheduler.TaskSetManager: Lost task 1.3 in stage 1.0 (TID 9) on executor <IP_ADDRESS>: org.apache.spark.util.TaskCompletionListenerException (Found unrecoverable error [Bad Request(4
00) - Invalid JSON fragment received[[1,"1"]][MapperParsingException[failed to parse]; nested: ElasticsearchParseException[Failed to derive xcontent from (offset=13, length=7): [123, 34, 105, 110, 100, 101
, 120, 34, 58, 123, 125, 125, 10, 91, 49, 44, 34, 49, 34, 93, 10, 123, 34, 105, 110, 100, 101

I looked at the elasticsearch logs and only see this cryptic message:
[2015-03-23 12:35:36,580][DEBUG][action.bulk              ] [Wizard] [test][0] failed to execute bulk item (index) index {[test][sparkes][AUxHfU4E8gA8zXpoJFVe], source[_na_]}
org.elasticsearch.index.mapper.MapperParsingException: failed to parse
  at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:563)
  at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:491)
  at org.elasticsearch.index.shard.service.InternalIndexShard.prepareCreate(InternalIndexShard.java:392)
  at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:444)
  at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:150)
  at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:512)
  at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:419)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  at java.lang.Thread.run(Thread.java:744)
Caused by: org.elasticsearch.ElasticsearchParseException: Failed to derive xcontent from (offset=76, length=7): [123, 34, 105, 110, 100, 101, 120, 34, 58, 123, 125, 125, 10, 91, 49, 44, 34, 49, 34, 93, 10,
 123, 34, 105, 110, 100, 101, 120, 34, 58, 123, 125, 125, 10, 91, 49, 44, 34, 49, 34, 93, 10, 123, 34, 105, 110, 100, 101, 120, 34, 58, 123, 125, 125, 10, 91, 49, 44, 34, 49, 34, 93, 10, 123, 34, 105, 110,
 100, 101, 120, 34, 58, 123, 125, 125, 10, 91, 49, 44, 34, 49, 34, 93, 10, 123, 34, 105, 110, 100, 101, 120, 34, 58, 123, 125, 125, 10, 91, 49, 44, 34, 49, 34, 93, 10, 123, 34, 105, 110, 100, 101, 120, 34,
 58, 123, 125, 125, 10, 91, 49, 44, 34, 49, 34, 93, 10, 123, 34, 105, 110, 100, 101, 120, 34, 58, 123, 125, 125, 10, 91, 49, 44, 34, 49, 34, 93, 10, 123, 34, 105, 110, 100, 101, 120, 34, 58, 123, 125, 125,
 10, 91, 49, 44, 34, 49, 34, 93, 10, 123, 34, 105, 110, 100, 101, 120, 34, 58, 123, 125, 125, 10, 91, 49, 44, 34, 49, 34, 93, 10, 123, 34, 105, 110, 100, 101, 120, 34, 58, 123, 125, 125, 10, 91, 49, 44, 34
, 49, 34, 93, 10, 123, 34, 105, 110, 100, 101, 120, 34, 58, 123, 125, 125, 10, 91, 50, 44, 34, 50, 34, 93, 10, 123, 34, 105, 110, 100, 101, 120, 34, 58, 123, 125, 125, 10, 91, 50, 44, 34, 50, 34, 93, 10, 1
23, 34, 105, 110, 100, 101, 120, 34, 58, 123, 125, 125, 10, 91, 50, 44, 34, 50, 34, 93, 10, 123, 34, 105, 110, 100, 101, 120, 34, 58, 123, 125, 125, 10, 91, 50, 44, 34, 50, 34, 93, 10, 123, 34, 105, 110, 1
00, 101, 120, 34, 58, 123, 125, 125, 10, 91, 50, 44, 34, 50, 34, 93, 10, 123, 34, 105, 110, 100, 101, 120, 34, 58, 123, 125, 125, 10, 91, 50, 44, 34, 50, 34, 93, 10, 123, 34, 105, 110, 100, 101, 120, 34, 5
8, 123, 125, 125, 10, 91, 50, 44, 34, 50, 34, 93, 10, 123, 34, 105, 110, 100, 101, 120, 34, 58, 123, 125, 125, 10, 91, 50, 44, 34, 50, 34, 93, 10, 123, 34, 105, 110, 100, 101, 120, 34, 58, 123, 125, 125, 1
0, 91, 50, 44, 34, 50, 34, 93, 10, 123, 34, 105, 110, 100, 101, 120, 34, 58, 123, 125, 125, 10, 91, 50, 44, 34, 50, 34, 93, 10, 123, 34, 105, 110, 100, 101, 120, 34, 58, 123, 125, 125, 10, 91, 51, 44, 34,
51, 34, 93, 10, 123, 34, 105, 110, 100, 101, 120, 34, 58, 123, 125, 125, 10, 91, 51, 44, 34, 51, 34, 93, 10, 123, 34, 105, 110, 100, 101, 120, 34, 58, 123, 125, 125, 10, 91, 51, 44, 34, 51, 34, 93, 10, 123
, 34, 105, 110, 100, 101, 120, 34, 58, 123, 125, 125, 10, 91, 51, 44, 34, 51, 34, 93, 10, 123, 34, 105, 110, 100, 101, 120, 34, 58, 123, 125, 125, 10, 91, 51, 44, 34, 51, 34, 93, 10, 123, 34, 105, 110, 100
, 101, 120, 34, 58, 123, 125, 125, 10, 91, 51, 44, 34, 51, 34, 93, 10, 123, 34, 105, 110, 100, 101, 120, 34, 58, 123, 125, 125, 10, 91, 51, 44, 34, 51, 34, 93, 10, 123, 34, 105, 110, 100, 101, 120, 34, 58,
 123, 125, 125, 10, 91, 51, 44, 34, 51, 34, 93, 10, 123, 34, 105, 110, 100, 101, 120, 34, 58, 123, 125, 125, 10, 91, 51, 44, 34, 51, 34, 93, 10, 123, 34, 105, 110, 100, 101, 120, 34, 58, 123, 125, 125, 10,
 91, 51, 44, 34, 51, 34, 93, 10, 123, 34, 105, 110, 100, 101, 120, 34, 58, 123, 125, 125, 10, 91, 52, 44, 34, 52, 34, 93, 10, 123, 34, 105, 110, 100, 101, 120, 34, 58, 123, 125, 125, 10, 91, 52, 44, 34, 52
, 34, 93, 10, 123, 34, 105, 110, 100, 101, 120, 34, 58, 123, 125, 125, 10, 91, 52, 44, 34, 52, 34, 93, 10, 123, 34, 105, 110, 100, 101, 120, 34, 58, 123, 125, 125, 10, 91, 52, 44, 34, 52, 34, 93, 10, 123,
34, 105, 110, 100, 101, 120, 34, 58, 123, 125, 125, 10, 91, 52, 44, 34, 52, 34, 93, 10, 123, 34, 105, 110, 100, 101, 120, 34, 58, 123, 125, 125, 10, 91, 52, 44, 34, 52, 34, 93, 10, 123, 34, 105, 110, 100,
101, 120, 34, 58, 123, 125, 125, 10, 91, 52, 44, 34, 52, 34, 93, 10, 123, 34, 105, 110, 100, 101, 120, 34, 58, 123, 125, 125, 10, 91, 52, 44, 34, 52, 34, 93, 10, 123, 34, 105, 110, 100, 101, 120, 34, 58, 1
23, 125, 125, 10, 91, 52, 44, 34, 52, 34, 93, 10, 123, 34, 105, 110, 100, 101, 120, 34, 58, 123, 125, 125, 10, 91, 52, 44, 34, 52, 34, 93, 10, 123, 34, 105, 110, 100, 101, 120, 34, 58, 123, 125, 125, 10, 9
1, 53, 44, 34, 53, 34, 93, 10, 123, 34, 105, 110, 100, 101, 120, 34, 58, 123, 125, 125, 10, 91, 53, 44, 34, 53, 34, 93, 10, 123, 34, 105, 110, 100, 101, 120, 34, 58, 123, 125, 125, 10, 91, 53, 44, 34, 53,
34, 93, 10, 123, 34, 105, 110, 100, 101, 120, 34, 58, 123, 125, 125, 10, 91, 53, 44, 34, 53, 34, 93, 10, 123, 34, 105, 110, 100, 101, 120, 34, 58, 123, 125, 125, 10, 91, 53, 44, 34, 53, 34, 93, 10, 123, 34
, 105, 110, 100, 101, 120, 34, 58, 123, 125, 125, 10, 91, 53, 44, 34, 53, 34, 93, 10, 123, 34, 105, 110, 100, 101, 120, 34, 58, 123, 125, 125, 10, 91, 53, 44, 34, 53, 34, 93, 10, 123, 34, 105, 110, 100, 10
1, 120, 34, 58, 123, 125, 125, 10, 91, 53, 44, 34, 53, 34, 93, 10, 123, 34, 105, 110, 100, 101, 120, 34, 58, 123, 125, 125, 10, 91, 53, 44, 34, 53, 34, 93, 10, 123, 34, 105, 110, 100, 101, 120, 34, 58, 123
, 125, 125, 10, 91, 53, 44, 34, 53, 34, 93, 10]

Has anyone solved this? Similar problem saving an RDD to ES with saveToEs
Similar exception and logs as yanakad points out.

Seems to be a duplicate of #403

@yanakad @aalkilani Something's amiss here. After getting side-tracked with some class-loading issue, I created a slightly different test case that passes just fine.
val schemaRDD = artistsAsBasicSchemaRDD
schemaRDD.saveAsParquetFile(outputParquet)

sqc.parquetFile(outputParquet).registerTempTable("testparquet")
val select = sqc.sql("SELECT * FROM testparquet")
println(select.schema)
select.saveToEs("test/parquet")

yields the following output:
20:33:36,314  INFO Executor task launch worker-0 hadoop.ParquetOutputFormat - Parquet block size to 134217728
20:33:36,314  INFO Executor task launch worker-0 hadoop.ParquetOutputFormat - Parquet page size to 1048576
20:33:36,314  INFO Executor task launch worker-0 hadoop.ParquetOutputFormat - Parquet dictionary page size to 1048576
20:33:36,314  INFO Executor task launch worker-0 hadoop.ParquetOutputFormat - Dictionary is on
20:33:36,314  INFO Executor task launch worker-0 hadoop.ParquetOutputFormat - Validation is off
20:33:36,315  INFO Executor task launch worker-0 hadoop.ParquetOutputFormat - Writer version is: PARQUET_1_0
20:33:36,324  WARN Executor task launch worker-0 zlib.ZlibFactory - Failed to load/initialize native-zlib library
20:33:36,324  INFO Executor task launch worker-0 compress.CodecPool - Got brand-new compressor
20:33:36,521  INFO Executor task launch worker-0 hadoop.InternalParquetRecordWriter - Flushing mem columnStore to file. allocated memory: 33,808,685
20:33:36,573  INFO Executor task launch worker-0 hadoop.ColumnChunkPageWriteStore - written 575B for [id] INT32: 345 values, 1,380B raw, 539B comp, 1 pages, encodings: [BIT_PACKED, PLAIN]
20:33:36,574  INFO Executor task launch worker-0 hadoop.ColumnChunkPageWriteStore - written 3,026B for [name] BINARY: 345 values, 5,144B raw, 2,983B comp, 1 pages, encodings: [BIT_PACKED, PLAIN]
20:33:36,574  INFO Executor task launch worker-0 hadoop.ColumnChunkPageWriteStore - written 3,472B for [url] BINARY: 345 values, 14,131B raw, 3,374B comp, 1 pages, encodings: [BIT_PACKED, PLAIN, RLE]
20:33:36,574  INFO Executor task launch worker-0 hadoop.ColumnChunkPageWriteStore - written 2,186B for [pictures] BINARY: 345 values, 18,395B raw, 2,060B comp, 1 pages, encodings: [BIT_PACKED, PLAIN, RLE]
20:33:36,589  INFO Executor task launch worker-0 output.FileOutputCommitter - Saved output of task 'attempt_201503262033_0005_r_000000_0' to file:...keyvaluerdd.parquet
...
StructType(List(StructField(id,IntegerType,false), StructField(name,StringType,false), StructField(url,StringType,true), StructField(pictures,StringType,true)))
...
== Physical Plan ==
ParquetTableScan [id#12,name#13,url#14,pictures#15], (ParquetRelation keyvaluerdd.parquet, Some(Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml), org.apache.spark.sql.SQLContext@7d74d959, []), []), which has no missing parents
...
20:33:37,089  INFO Executor task launch worker-0 rdd.NewHadoopRDD - Input split: ParquetInputSplit{part: file:...keyvaluerdd.parquet/part-r-1.parquet start: 0 end: 10097 length: 10097 hosts: [] requestedSchema: message root {
  required int32 id;
  required binary name (UTF8);
  optional binary url (UTF8);
  optional binary pictures (UTF8);
}
 readSupportMetadata: {org.apache.spark.sql.parquet.row.metadata={"type":"struct","fields":[{"name":"id","type":"integer","nullable":false,"metadata":{}},{"name":"name","type":"string","nullable":false,"metadata":{}},{"name":"url","type":"string","nullable":true,"metadata":{}},{"name":"pictures","type":"string","nullable":true,"metadata":{}}]}, org.apache.spark.sql.parquet.row.requested_schema={"type":"struct","fields":[{"name":"id","type":"integer","nullable":false,"metadata":{}},{"name":"name","type":"string","nullable":false,"metadata":{}},{"name":"url","type":"string","nullable":true,"metadata":{}},{"name":"pictures","type":"string","nullable":true,"metadata":{}}]}}}
20:33:37,093  WARN Executor task launch worker-0 hadoop.ParquetRecordReader - Can not initialize counter due to context is not a instance of TaskInputOutputContext, but is org.apache.hadoop.mapreduce.TaskAttemptContext
20:33:37,115  INFO Executor task launch worker-0 hadoop.InternalParquetRecordReader - RecordReader initialized will read a total of 345 records.
...

That is, a schema RDD is saved to a parquet file, read back, registered as a table and its result indexed to ES after printing the schema. Everything works fine.
That is not to say that the bug doesn't occur in your rather I can't seem to reproduce it - can you please enable logging or try out the example above (it's also in master)

@costin do you have instructions on how to run the test. I did hte following:
git clone https://github.com/elastic/elasticsearch-hadoop.git
Open IntelliJ (13.1.5 community edition)
Import project, select build.gradle file
when I try to run the test I get the following error:
Error:(4, 25) object sql is not a member of package org.apache.spark
import org.apache.spark.sql.SQLContext
                        ^

spark-sql shows as a test dependency but not a compile one, not sure what's going on
One thing that I noticed is perhaps different is that I am using a CDH4 distribution of spark. I'll try the plain hadoop distro tonight but wanted to see if you have any tips on how to run the test as you have it.
My spark side logs seem fine:
15/03/25 16:30:14 WARN ParquetRecordReader: Can not initialize counter due to context is not a instance of TaskInputOutputContext, but is org.apache.hadoop.mapreduce.ta
sk.TaskAttemptContextImpl
15/03/25 16:30:14 WARN ParquetRecordReader: Can not initialize counter due to context is not a instance of TaskInputOutputContext, but is org.apache.hadoop.mapreduce.ta
sk.TaskAttemptContextImpl
15/03/25 16:30:14 INFO NewHadoopRDD: Input split: ParquetInputSplit{part: file:/home/ykadiysk/Github/spark_1.2.1/bin/large.parquet/part-r-3.parquet start: 0 end: 554 le
ngth: 554 hosts: [] requestedSchema: message root {
  required int32 key;
  optional binary value (UTF8);
}
 readSupportMetadata: {org.apache.spark.sql.parquet.row.metadata={"type":"struct","fields":[{"name":"key","type":"integer","nullable":false,"metadata":{}},{"name":"valu
e","type":"string","nullable":true,"metadata":{}}]}, org.apache.spark.sql.parquet.row.requested_schema={"type":"struct","fields":[{"name":"key","type":"integer","nullab
le":false,"metadata":{}},{"name":"value","type":"string","nullable":true,"metadata":{}}]}}}
15/03/25 16:30:14 WARN ParquetRecordReader: Can not initialize counter due to context is not a instance of TaskInputOutputContext, but is org.apache.hadoop.mapreduce.ta
sk.TaskAttemptContextImpl
15/03/25 16:30:14 WARN ParquetRecordReader: Can not initialize counter due to context is not a instance of TaskInputOutputContext, but is org.apache.hadoop.mapreduce.ta
sk.TaskAttemptContextImpl
15/03/25 16:30:14 INFO InternalParquetRecordReader: RecordReader initialized will read a total of 12 records.
15/03/25 16:30:14 INFO InternalParquetRecordReader: RecordReader initialized will read a total of 13 records.
15/03/25 16:30:14 INFO InternalParquetRecordReader: RecordReader initialized will read a total of 13 records.
15/03/25 16:30:14 INFO InternalParquetRecordReader: at row 0. reading next block
15/03/25 16:30:14 INFO CodecPool: Got brand-new decompressor [.gz]
15/03/25 16:30:14 INFO InternalParquetRecordReader: block read in memory in 0 ms. row count = 12
15/03/25 16:30:14 INFO InternalParquetRecordReader: RecordReader initialized will read a total of 12 records.
15/03/25 16:30:14 INFO EsRDDWriter: Writing to [test/spark]
15/03/25 16:30:14 INFO EsRDDWriter: Writing to [test/spark]
15/03/25 16:30:14 INFO EsRDDWriter: Writing to [test/spark]
15/03/25 16:30:27 INFO InternalParquetRecordReader: at row 0. reading next block
15/03/25 16:30:27 INFO InternalParquetRecordReader: at row 0. reading next block
15/03/25 16:30:27 INFO CodecPool: Got brand-new decompressor [.gz]
15/03/25 16:30:27 INFO InternalParquetRecordReader: block read in memory in 1 ms. row count = 12
15/03/25 16:30:27 INFO CodecPool: Got brand-new decompressor [.gz]
15/03/25 16:30:27 INFO InternalParquetRecordReader: block read in memory in 14 ms. row count = 13
15/03/25 16:30:27 INFO InternalParquetRecordReader: at row 0. reading next block
15/03/25 16:30:27 INFO CodecPool: Got brand-new decompressor [.gz]
15/03/25 16:30:27 INFO InternalParquetRecordReader: block read in memory in 0 ms. row count = 13
15/03/25 16:31:42 ERROR TaskContextImpl: Error in TaskCompletionListener
org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest: Found unrecoverable error [Bad Request(400) - Invalid JSON fragment received[[26,"26"]][MapperParsingException[failed to parse]; nested: ElasticsearchParseException[Failed to derive xcontent from (offset=13, length=9): [123,

So, I guess what I'm seeing is a disparity between the work that users are doing on their own and the code testing that's in place.
The response seems to be "tests work fine, must be you".  I can accept this because I probably am doing something wrong, but now that we've had multiple people mention some issues I think it makes sense to step back and look at what the that difference is between what us users are doing and how the test environment/process works.
As a user I'd like to download this code base, package it, start up spark code, and execute.  Unfortunately, that doesn't seem to work as we assume it should.
What can we do?

@costin @yanakad @mshirley
In my case the test was even simpler. To summarize:
Dependencies:
elasticsearch-spark_2.10 Version: 2.1.0.Beta3
spark-streaming_2.10 Version: 1.2.1
Running on:
Yarn HDP cluster
Test #1: Save RDD[Map[String,Int]]. Result: Pass. This works
Test #2: Save RDD[String] where String is a JSON string. This fails.
Test #3: Save RDD[Case Class] According to documentation this should work. It fails as well.
Here's the documentation I'm referring to
http://www.elastic.co/guide/en/elasticsearch/hadoop/master/spark.html
Using the exact same example does not work. (The documentation also has a typo naming the variable +RDD+). Here's the example. Again, this did not work and threw an exception.
// define a case class
case class Trip(departure: String, arrival: String)
val upcomingTrip = Trip("OTP", "SFO")
val lastWeekTrip = Trip("MUC", "OTP")
val rdd = sc.makeRDD(Seq(upcomingTrip, lastWeekTrip))
EsSpark.saveToEs(rdd, "spark/docs")
And the exception was:
org.apache.spark.util.TaskCompletionListenerException: Found unrecoverable error [Bad Request(400) - Invalid JSON fragment received[["MUC","OTP"]][MapperParsingException[failed to parse]; nested: ElasticsearchParseException[Failed to derive xcontent from (offset=0, length=13): [91, 34, 77, 85, 67, 34, 44, 34, 79, 84, 80, 34, 93]]; ]]; Bailing out..
at org.apache.spark.TaskContextImpl.markTaskCompleted(TaskContextImpl.scala:76)
at org.apache.spark.scheduler.Task.run(Task.scala:58)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
The example couldn't really get any simpler than that. If I manually create a Map out of the values in the case class, it works. This example even removes the SQL context schema out of the equation.
Thoughts?

@aalkilani this is very odd. That example DID work for me. Being new to ES I tested this right after I first saw the error...

@yanakad See the readme. IDEs tend to be picky and the command line tends to work best across environments.
The readme has links to two CI builds - Travis and Jenkins to monitor the test build across different machines. It looks there's an issue when running the parquet test against Hadoop 1.x so I'll try to switch it to Hadoop 2.x..
To run the spark test, you can simply do from the root folder:
gradlew elasticsearch-spark:integrationTest

To clarify, are you using Spark 1.2.1 the CDH binary on your own machines or against a certain Hadoop distro? Also can you double/triple-check that the latest/same es-hadoop version is used across all your nodes?
I mention this since the case class support has been added several weeks ago - if you have an older snapshot which might be still be laying around, the case class will be serialized as a Product and as such, its structure will not be recognized.

@mshirley I cannot reproduce the issue as I've mentioned several times in this thread - the tests in master have been updated and the CI builds are passing. Clearly there's a difference somewhere and that's why I've been asking for increased logs (including from you) - without them, I can only guess what is wrong.

what can we do?

As I've said before, enable logging on the rest package (see this section) and provide accurate information about your environment (as mentioned in the troubleshoot section) ).
The more information there is about failing environment, the better.
And always, use the latest snapshot (maven, gradle, sbt all work fine with it).

@aalkilani I've added minor formatting to your post to make it readable.
It looks like you have some configuration issues.
Test #2 doesn't work unless you tell es-hadoop that the RDD contains JSON. This is explain in the docs - basically use saveJsonToEs method.
There's no way (that I know of) to bind saveToEs to all RDDs except those on type String or Array[byte] hence why you need to use the dedicated method.
Test 3: fails since you are using the incorrect version. You are using Beta3 but the case class feature is part of master (the dev build). Hence why the master docs contain this section but the 2.1Beta ones do not.

@costin, I was able to repro yesterday with the plain hadoop distro as well (no cluster, just download, run spark-shell in local mode -- takes about 10min on a clean machine)
wget http://www.eng.lsu.edu/mirrors/apache/spark/spark-1.2.1/spark-1.2.1-bin-hadoop2.3.tgz
untar
cd spark-1.2.1-bin-hadoop2.3/bin
wget https://oss.sonatype.org/content/repositories/snapshots/org/elasticsearch/elasticsearch-hadoop/2.1.0.BUILD-SNAPSHOT/elasticsearch-hadoop-2.1.0.BUILD-20150324.023417-341.jar
 ./spark-shell --jars elasticsearch-hadoop-2.1.0.BUILD-20150324.023417-341.jar

code from issue 403

I will install gradle and see if I can get your passing test

@yanakad Can you upload the failing scala test into a gist? I should be able to reproduce it from the command-line then.

@costin this should be a very clean + easy setup (everything is local -- Spark and ES, no clusters)
https://gist.github.com/yanakad/b7ed4d799f38c3ca4282
Im seeing failures on the gradle tests but I won't distract you with this for now -- prob easier if you get a repro

@yanakad thanks for the gist. I've been able to reproduce the error using the command line. Will keep you posted once I figure out what's wrong; the SchemaRDD looks fine and I suspect it might be the Parquet implementation that throws the parser off.

@costin
OS & JVM version:
CentOS 6.6 Final
java version "1.7.0_75"
OpenJDK Runtime Environment (rhel-<IP_ADDRESS>.el6_6-x86_64 u75-b13)
OpenJDK 64-Bit Server VM (build 24.75-b04, mixed mode)
Hadoop / Spark version / distribution:
hadoop-2.6.0
spark-1.2.1-bin-hadoop2.4
elasticsearch-hadoop version:
elasticsearch-hadoop-2.1.0.BUILD-20150324.023417-341
Hadoop / Spark cluster size:
1
Elasticsearch cluster size:
1
The size of the dataset and a snippet of it in its raw format (CSV, TSV, etc..):
16 bytes
Gist for workflow on a fresh centos 6.6 vm:
https://gist.github.com/mshirley/88e5b0b94be38b3e1d77
Gist of spark-shell errors:
https://gist.github.com/mshirley/efb3f8f038d34b6dbaa7
Gist of elasticsearch errors:
https://gist.github.com/mshirley/6bb178826b83e04199b8
Snippet showing the data being sent to ES:
15/03/27 21:18:20 DEBUG HttpConnection: Open connection to <IP_ADDRESS>:9200
15/03/27 21:18:20 DEBUG header: >> "PUT /spark_test/test/_bulk HTTP/1.1[\r][\n]"
15/03/27 21:18:20 DEBUG HttpMethodBase: Adding Host request header
15/03/27 21:18:20 DEBUG header: >> "User-Agent: Jakarta Commons-HttpClient/3.1[\r][\n]"
15/03/27 21:18:20 DEBUG header: >> "Host: <IP_ADDRESS>:9200[\r][\n]"
15/03/27 21:18:20 DEBUG header: >> "Content-Length: 23[\r][\n]"
15/03/27 21:18:20 DEBUG header: >> "Content-Type: application/json; charset=UTF-8[\r][\n]"
15/03/27 21:18:20 DEBUG header: >> "[\r][\n]"
15/03/27 21:18:20 DEBUG content: >> "{"index":{}}[\n]"
15/03/27 21:18:20 DEBUG content: >> "["value"][\n]"

@yanakad @mshirley I found the problem - you are both using the wrong package.
To save a SchemaRDD one needs to import org.elasticsearch.spark.sql._ not org.elasticsearch.spark._.
The former binds saveToEs to a SchemaRDD (a special kind of RDD) while the latter binds it to any kind of RDD. Since a SchemaRDD is an RDD, the latter will work but will disregard its schema resulting in arrays which are invalid JSON documents.
Going forward I'm thinking of using a different name for saveToEs to avoid this type of mistake in the future. In Spark 1.3 the DataFrame is not an RDD any more so this type of binding will not occur any more; on the other hand users of 1.1 and 1.2 might still fall into this trap.

@costin Confirmed working!  Thanks!
changed:
import org.elasticsearch.spark._

to:
import org.elasticsearch.spark.sql._

I'm reopening to address this somehow. I got lost in this as well and I suspect other will too - I'm planning on adding a warning so if somebody uses pure saveToEs on a SchemaRDD they get notified.

@costin agreed -- a warning would be great. At a very minimum a second example/special note in the Spark section of the README would help. It's a pretty subtle difference in the import and I suspect most people like myself would miss it (for example I'm aware that a SchemaRDD is a special kind of RDD but it would not have occurred to me that I need to import something different to get the correct behavior).
Thanks again for looking into this.

@yanakad @mshirley I've added a warning in master. If the method is called on a SchemaRDD, a warning will be displayed.
Note though that master has moved to Spark 1.3 which is not backward with 1.2 (on the SQL package). We do provide however a dedicated jar for it (namely elasticsearch-spark-1.2-XXXX).
Cheers,

@costin
Thanks for all your help!


Add support for Drupal Console custom packages and languages

The Drupal Console project is working in complete some goals to release his first 1.0.0 stable release more info at https://drupalconsole.com/articles/help-us-complete-drupal-console-stable-release
One of the main component required for that its provide a way to Drupal developer to be able to add custom packages of commands to be used in their Drupal installation, independent of contributed modules command
Another goal for this release its allows the usage of custom languages, that will reduce the size of Drupal Console installer and people will be freer in terms of their contribution and how to use the Drupal Console project.
In order to warranty this flexibility is required the inclusion of two new composers installers types drupal-console for packages and drupal-console-languages for languages.

👍  For this PR.

Drupal console will become the defacto command line management tool for Drupal. This is very important for supporting the Drupal project.

This PR has merge conflicts.

Thanks @ofry , I just updated the PR

@alcohol travis is failing, could due to the latest change?

@enzolutions no I already pointed out why it fails, see my comment about the missing trailing comma.


Proxy settings are not used in the code

Ably Kafka connector version: 1.0.2
I tried to find the root case of my issues with the Ably Kafka connector and found out that the proxy config is not used in the client options object:
    opts.recover = getString(CLIENT_RECOVER);
    if (getBoolean(CLIENT_PROXY)) {
      ProxyOptions proxyOpts = new ProxyOptions();
      proxyOpts.host = getString(CLIENT_PROXY_HOST);
      proxyOpts.port = getInt(CLIENT_PROXY_PORT);
      proxyOpts.username = getString(CLIENT_PROXY_USERNAME);
      proxyOpts.password = getPassword(CLIENT_PROXY_PASSWORD).value();
      proxyOpts.nonProxyHosts = getList(CLIENT_PROXY_NON_PROXY_HOSTS).toArray(new String[0]);
      proxyOpts.prefAuthType = HttpAuth.Type.valueOf(getString(CLIENT_PROXY_PREF_AUTH_TYPE));
    }
    opts.environment = getString(CLIENT_ENVIRONMENT);

I expect a line like
opts.proxy = proxyOpts;

at the end of the if block above but don't see it, unfortunately.
┆Issue is synchronized with this Jira Uncategorised by Unito

Thanks for reporting and sorry for the late response. Your analysis looks correct, we just need to validate that the underlying Java client library supports these options.

This issue has been resolved in Version 3 of the connector :)


Fixed the exponential interpolator

Fixed the exponential interpolator.
https://github.com/avhz/RustQuant/pull/162#issuecomment-1926279538
One of the changes was in the actual computation to the exponential interpolation. I'm not too sure how your method previously worked, but the following method is quite straightforward, and it seems to fix the issue.
Method:
0. Starting with (x_1, y_1) and (x_2, y_2) and we want to interpolate at some point x_1 < p < x_2.

Take the log of the y-values. That is, we now have (x_1, ln(y_1)) and (x_2, ln(y_2)).
Use linear interpolation between these two points.
Apply exp to it to convert back out of logspace.

Awesome, thanks for this one !


I cannot get it to work with a Zebra iMZ320 printer and Android 9.0. The printer is not listed when I scan for devices. Has anyone achieved it yet?

For Zebra printers you should use Zebra SDK, this just works for thermal
bluetooth printers compatible with esc/pos standard.
On Tue, Apr 21, 2020, 6:59 AM Michail Papadakis<EMAIL_ADDRESS>wrote:

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/CrossGeeks/BluetoothPrintSample/issues/1, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/ABKWUVRDBKYZEF32JOWR7P3RNV37TANCNFSM4MNGQDXA
.

For Zebra printers you should use Zebra SDK, this just works for thermal bluetooth printers compatible with esc/pos standard.
…
On Tue, Apr 21, 2020, 6:59 AM Michail Papadakis @.***> wrote: — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABKWUVRDBKYZEF32JOWR7P3RNV37TANCNFSM4MNGQDXA .

I have already implemented it with Zebra SDK but a universal printing service for all bluetooth printers would be great!


Cannot read property 'my_property' of null (Use dot notation)

Hi,
First of all, thank for your library. It's awesome!
I've tried to make my own template to generate an excel but when I write something like 'data.otherProperty' it throws an error like "Cannot read 'otherProperty' property of null" when 'data' does not exist. I prefer access to data with dot notation and maybe it could be more comfortable
I overwrite your code and this let me generate an excel without errors:
`
// In your /app/node_modules/xlsx-template-ex/dist/app/template-engine.js file
const R = require('ramda') // or lodash
........
processTplValues(tplExp, data) {
// if (!resultData) {
//     resultData = tplExp.valueName.split(".").reduce((o, key) => ((o && o[key]) || o[key] === 0 ? o[key] : null), this.data);
// }
return R.pathOr('',tplExp.valueName.split('.'),data);
}
`
Maybe you have to implement it in other cases like pipes. And if you could add nested arrays access with dot notation...
I hope you could add this change and make a release to download.
Thank you!

@RBN1993
Hello! Check examples with custom pipes:
https://github.com/optimistex/xlsx-template-ex/blob/cb57afaed602cb075af4a25600f34199e6b98a0a/src/example.ts#L285-L286
https://github.com/optimistex/xlsx-template-ex/blob/cb57afaed602cb075af4a25600f34199e6b98a0a/src/example.ts#L312-L318

Most probably you could add your own pipe that reads nested properties by your own rules.
Also, I highly recommend to prepare/simplify your data before render. For instance, when you need to render a deeply nested property it's a sign that you have not so well prepared data. Process it before render. My main idea to keep the template engine as simple as it possible and ensure Excel things like insert a picture or clone cells, but not to invent a data engine. For processing data you already have NodeJs. So, use NodeJs to bring your complex data set to some simple representation that is ready to be rendered into an Excel.


Sass partial changes are not detected

I am including sass partials on a main.scss file with
@import "colors", "footer", "header";

The sass file is then compiled with
{% block stylesheets %}
        {% stylesheets filter="compass" filter="?uglifycss"
<EMAIL_ADDRESS>            output='css/evossc.css'
        %}
            <link rel="stylesheet" href="{{ asset_url }}" />
        {% endstylesheets %}
    {% endblock %}

But when I modify any of the partials the change is not reflected on Symfony. Only changing the main file works. I ve tried to put all the partials within the stylesheets tag instead, but it this way sass variables are not seen by other files.

does it work if you use separate @import rules for each partial instead of putting them all in the same rule ?
Recent Assetic versions (I don't remember when it was implemented for Sass, so I cannot say what "recent" implies exactly) should support taking imported files into account for the detection of changes. but I don't think it supports multiple imports at once (I was not even aware this syntax was possible)

I've separated the rules but the problem remains. Based on the assetic repo I've also see that this should be fixed both for sass and compass filters , I am using the dev-master versions but still no luck...

Possibly related to #332. At least, the symptoms are the same.

Any idea  ? Can #332 be merged ?

#332 still needs more review before it can be merged, but I do believe it would solve your problem.

#332 has been merged and is part of the 2.7.0 release


Replace Device Classes with SensorDeviceClass

Device classes are deprecated. We need to replace them with 'SensorDeviceClass' before 2025.
DEVICE_CLASS_AQI: Final = "aqi"
DEVICE_CLASS_BATTERY: Final = "battery"
DEVICE_CLASS_CO: Final = "carbon_monoxide"
DEVICE_CLASS_CO2: Final = "carbon_dioxide"
DEVICE_CLASS_CURRENT: Final = "current"
DEVICE_CLASS_DATE: Final = "date"
DEVICE_CLASS_ENERGY: Final = "energy"
DEVICE_CLASS_FREQUENCY: Final = "frequency"
DEVICE_CLASS_GAS: Final = "gas"
DEVICE_CLASS_HUMIDITY: Final = "humidity"
DEVICE_CLASS_ILLUMINANCE: Final = "illuminance"
DEVICE_CLASS_MONETARY: Final = "monetary"
DEVICE_CLASS_NITROGEN_DIOXIDE = "nitrogen_dioxide"
DEVICE_CLASS_NITROGEN_MONOXIDE = "nitrogen_monoxide"
DEVICE_CLASS_NITROUS_OXIDE = "nitrous_oxide"
DEVICE_CLASS_OZONE: Final = "ozone"
DEVICE_CLASS_PM1: Final = "pm1"
DEVICE_CLASS_PM10: Final = "pm10"
DEVICE_CLASS_PM25: Final = "pm25"
DEVICE_CLASS_POWER_FACTOR: Final = "power_factor"
DEVICE_CLASS_POWER: Final = "power"
DEVICE_CLASS_PRESSURE: Final = "pressure"
DEVICE_CLASS_SIGNAL_STRENGTH: Final = "signal_strength"
DEVICE_CLASS_SULPHUR_DIOXIDE = "sulphur_dioxide"
DEVICE_CLASS_TEMPERATURE: Final = "temperature"
DEVICE_CLASS_TIMESTAMP: Final = "timestamp"
DEVICE_CLASS_VOLATILE_ORGANIC_COMPOUNDS = "volatile_organic_compounds"
DEVICE_CLASS_VOLTAGE: Final = "voltage"

DEVICE_CLASS_TEMPERATURE -> SensorDeviceClass.TEMPERATURE
DEVICE_CLASS_TEMPERATURE -> SensorDeviceClass.HUMIDITY

Ciao,
posso farlo io per darti una mano?
Se mi dici dove modificare mi ci metto io.

Hi, this is a WIP.
I have been doing some work. I need to finish them, but I have been too busy. I apologise.
Anyway, there is a branch for this. So you can look at it.

I have made the change. Code is in master. Will be tagged after some feedback.

Close as code has been merged


Fix for cryptography library dependency

Status
Ready/In Progress/In Hold (Reason for hold)
Related Issues
related: link to the issue
Description
A few sentences describing the overall goals of the pull request's commits.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

@lifehackjim thanks for your contribution. Do you want to also updated the version of axonius_api_client?

@lifehackjim thanks for your contribution. Do you want to also updated the version of axonius_api_client?

@lifehackjim thanks for your contribution. Do you want to also updated the version of axonius_api_client?

Actually, yes - thank you for reminding me.
Is it permitted for me to do axonius-api-client = "==4.*" in your Pipfiles?

@lifehackjim thanks for your contribution. Do you want to also updated the version of axonius_api_client?

Actually, yes - thank you for reminding me.
Is it permitted for me to do axonius-api-client = "==4.*" in your Pipfiles?

I switched from using pipenv to normal requirements.txt since pipenv is centered around locking to specific versions where I want the latest minor or micro release of the v4 major version

I switched from using pipenv to normal requirements.txt since pipenv is centered around locking to specific versions where I want the latest minor or micro release of the v4 major version

@lifehackjim No problem to use Pipenv to limit to 4.x. I've opened a separate PR for this: https://github.com/demisto/dockerfiles/pull/3678. We will manage the update in that PR. Also we've updated the base Crypto image, so no need for changes in the dev dependencies.

@lifehackjim No problem to use Pipenv to limit to 4.x. I've opened a separate PR for this: https://github.com/demisto/dockerfiles/pull/3678. We will manage the update in that PR. Also we've updated the base Crypto image, so no need for changes in the dev dependencies.

Closing as this has been updated in https://github.com/demisto/dockerfiles/pull/3678

Closing as this has been updated in https://github.com/demisto/dockerfiles/pull/3678


Add 'binding' to all binding package names

The maven groupId should be equal to the prefix of the package name. So we should add binding to all of our binding package names.

This issue is fixed due PR #80


Circuit's QASM output doesn't contain stdgates.inc include

Describe the bug
Exactly as the title says.
from braket.circuits import Circuit
from braket.circuits.serialization import IRType

bell = Circuit().h(0).cnot(0, 1)
qasm = bell.to_ir(IRType.OPENQASM)
print(qasm.source)

OPENQASM 3.0;
bit[2] b;
qubit[2] q;
h q[0];
cnot q[0], q[1];
b[0] = measure q[0];
b[1] = measure q[1];

This causes a problem for any arbitrary OQ3 parser, because stdgates.inc is not declared, which means the h and cnot are undefined gates.
Expected behavior
The QASM output should contain include "stdgates.inc";.

Hi, this is the expected behavior. Circuit gates are translated to built-in Braket gates, such as h and cnot, which are understood by Braket backends like the local and on-demand simulators. These built-in gates differ from the ones defined in stdgates.inc, with some names being changed and some differences in which gates are present. To work with an arbitrary OQ3 backend, you would need to supply an implementation of the Braket gates in OpenQASM. I am curious to hear more about your use case; feel free to reference braket_gates.inc to see some reference implementations of our built-in gates.

I see, thank you for the pointer to braket_gates.inc. I'm just writing my own general OQ3 parser, and happened to encounter the gates include issue. Should I close this issue, or are you planning to add explicit include "braket_gates.inc"; in the QASM output?

We don't currently have plans to add that include statement, since those gates are implicitly understood by all Braket backends. You should be able to create a translator function that takes a Braket Circuit OQ3 output and transforms it into a general form with the inclusion of the braket_gates.inc file. The default simulator oq3 implementation may be a useful resource to reference! Please let me know if you have any questions, otherwise feel free to close this issue

I see, thank you for the pointer to the simulator. If I have more questions, I will create a separate issue instead.


python3.pkgs: opentelemetry-api: 1.21.0 -> 1.22.0, opentelemetry-instrumentation: 1.16.0 -> 0.43b0 (sic), opentelemetry-instrumentation-flask: init at 0.43b0

Description of changes
Update opentelemetry-python (and -contrib) to latest versions available.
cc @mweinelt I guess the python package updater needs some fixup for opentelemetry-instrumentation (or just exclude it), see c913a0bc0da401762630277afe845ff2cf8289a3 for context.
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

Result of nixpkgs-review pr 282349 at f88ea9b9 run on x86_64-linux 1

15 packages marked as broken and skipped:

python311Packages.bentoml
python312Packages.opentelemetry-api
python312Packages.opentelemetry-exporter-otlp
python312Packages.opentelemetry-exporter-otlp-proto-common
python312Packages.opentelemetry-exporter-otlp-proto-grpc
python312Packages.opentelemetry-exporter-otlp-proto-http
python312Packages.opentelemetry-exporter-prometheus
python312Packages.opentelemetry-instrumentation
python312Packages.opentelemetry-instrumentation-asgi
python312Packages.opentelemetry-instrumentation-flask
python312Packages.opentelemetry-instrumentation-grpc
python312Packages.opentelemetry-instrumentation-wsgi
python312Packages.opentelemetry-sdk
python312Packages.opentelemetry-test-utils
python312Packages.opentelemetry-util-http

3 packages failed to build:

apache-airflow
python311Packages.qcodes
python311Packages.qcodes-contrib-drivers

21 packages built successfully:

python311Packages.opentelemetry-api
python311Packages.opentelemetry-exporter-otlp
python311Packages.opentelemetry-exporter-otlp-proto-common
python311Packages.opentelemetry-exporter-otlp-proto-grpc
python311Packages.opentelemetry-exporter-otlp-proto-http
python311Packages.opentelemetry-exporter-prometheus
python311Packages.opentelemetry-instrumentation
python311Packages.opentelemetry-instrumentation-aiohttp-client
python311Packages.opentelemetry-instrumentation-asgi
python311Packages.opentelemetry-instrumentation-django
python311Packages.opentelemetry-instrumentation-flask
python311Packages.opentelemetry-instrumentation-grpc
python311Packages.opentelemetry-instrumentation-wsgi
python311Packages.opentelemetry-proto
python311Packages.opentelemetry-sdk
python311Packages.opentelemetry-semantic-conventions
python311Packages.opentelemetry-test-utils
python311Packages.opentelemetry-util-http
python311Packages.strawberry-graphql
python312Packages.opentelemetry-proto
python312Packages.opentelemetry-semantic-conventions


Error when creating a group under the root group.

If you want to createe a group under the root group you get an error. But in the keepass file the group is created.
https://github.com/Torie-Coding/keepass/blob/44df94be9cc4e7072d1281d091c06d8c21a81a27/plugins/modules/group.py#L253-L256

Merged into dev. Will be in next release


qtile: move to python-modules

Description of changes
Since qtile is really a Python application, it belongs in python-modules. Thanks to this move the current system Python isn't pinned in the package anymore, so it can be built on multiple Python versions.
This fixes the ZHF build error of python3Packages.qtile-extras on non-system Python versions.
Big thanks to @NobbZ for figuring this out and providing the patch!
ZHF: #230712

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[x] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[x] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
23.05 Release Notes (or backporting 22.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Since qtile is really a Python application, it belongs in python-modules.

that is not quite true, only python library belong into pythonPackages because we loose freedom to overwrite dependencies however we like.

My bad. Actually qtile is both a library and an application. Case in point: qtile-extras imports libqtile.

My bad. Actually qtile is both a library and an application.

If it uses buildPythonApplication then it goes into python-modules, otherwise outside and gets converted into a pythonPackages inside of python-modules. Either way it has an entry in both pkgs. and pkgs.pythonPackages.

Case in point: qtile-extras imports libqtile.

I am not sure what you mean with that.

Thank you again for helping me! I've addressed all your comments and tested the results (I'm running Qtile now).

@SuperSandro2000 do you think we can get this in for the ZHF?

Result of nixpkgs-review pr 231252 run on x86_64-linux 1

9 packages built:

python310Packages.qtile
python310Packages.qtile-extras
python310Packages.qtile-extras.dist
python310Packages.qtile.dist
python311Packages.qtile
python311Packages.qtile-extras
python311Packages.qtile-extras.dist
python311Packages.qtile.dist
qtile


Introduce Provider<T>.forUseAtConfigurationTime(): Provider<T>

So a new wrapper can be built with the new API and used to make the build compatible with the future implementation.

@bot-gradle test this

OK, I've already triggered ReadyForMerge build for you.


Collect trace data through the observatory HTTP interface

@johnmccutchan or @devoncarew are probably your best reviewers.  I checked and both web_socket_channel and json_rpc_2 were developed by the Dart team/Google, so there is no concern re: licenses of those dependencies.

lgtm, though I think we should simplify the file name generating; it'll be similar to other places where we create files.

lgtm!
@jason-simmons, my reading of this code now is that it's no longer android specific, and could work with iOS as well. Does that sounds right?

lgtm

my reading of this code now is that it's no longer android specific, and could work with iOS as well.

Yes! Thanks Jason.


Add an Alarm Feature

Add time with date.
when the time and date match with the current date and time an alert pops up with the task written on it.
and music will play with this alert as an alarm.
using JavaScript, HTML, CSS.

no need


Added FFT and Commented code to Math Algos

[x] Fast Fourier Transformation

[ ] Sum of squares of first N natural numbers

[x] Checking for integer overflow on multiplication

[ ] Convert Radian to Degree

[x] K-th Primes

I have commented the codes and also added READMEs

Sir, should I be working on more math algorithms? .

Sir, can i get a feedback ?

I think its good.
What do you say @soubh1k ?

I am merging this


Display data is now required

What needs to happen?
No longer default to SPHERE, error out if no display data
Why should we do this?
Simulariumio has been updated to have SPHERE GROUP which means we no longer have a default display_type
When does this need to get done?
Once the change is merged into simulariumio and a new version is released

this has been released in simulariumio

We used to use sphere as a proxy geometry in case agent data arrived and was ready before the geometry arrived.

Yeah, I'm looking at this now, and we do give a warning with there is no display type given before we default it to a sphere. I know I'm the one that wrote this, but I'm now not sure why we should do this. Ie, alerting the user that they're missing data, but still rendering SOMETHING seems to be a good behavior?

It seems like if display data is required data, (on the simulariumio side and here) then we're good to go!  We don't need to get rid of the sphere default, unless you really don't want to see agents whose geometry hasn't arrived yet or has errored out..

Yeah, so @blairlyons is it ok for me to not do anything here? We've already implemented SPHERE GROUP in the viewer. and again, we do warn users of their mistake if the data is missing.
export const makeMissingDisplayTypeErrorMessage = (
    key: string,
    url: string
): string => {
    if (url) {
        return `Missing typeMapping[${key}].geometry.displayType, so we couldn't request ${url}. Geometry will default to spheres`;
    } else {
        return `No typeMapping[${key}].geometry.displayType. Geometry will default to spheres`;
    }
};


Outdated dependencies

I saw the following while running npm install:
~/gh/solid/solid-panes $ npm install
npm WARN deprecated<EMAIL_ADDRESS>🙌  Thanks for using Babel: we recommend using babel-preset-env now: please read https://babeljs.io/env to update!
npm WARN deprecated<EMAIL_ADDRESS>core-js@<3 is no longer maintained and not recommended for usage due to the number of issues. Please, upgrade your dependencies to the actual version of core-js@3.
npm WARN deprecated<EMAIL_ADDRESS>This is a stub types definition. rdflib provides its own type definitions, so you do not need this installed.
npm WARN deprecated<EMAIL_ADDRESS>Use uuid module instead
npm WARN deprecated<EMAIL_ADDRESS>no longer maintained

Though probably not harmful, these look bad to new contributors. I'll see if I can follow up each of those 5 recommendations.

First three fixed.
Fourth continued in https://github.com/solid/solid-ui/issues/192
Fifth continued in https://github.com/solid/solid-panes/issues/219

@michielbdejong w.r.t.

npm WARN deprecated<EMAIL_ADDRESS>This is a stub types definition. rdflib provides its own type definitions, so you do not need this installed.

See https://github.com/linkeddata/rdflib.js/issues/369#issuecomment-581138694 and https://github.com/DefinitelyTyped/DefinitelyTyped/pull/42044. `@types/rdflib shouldn't be removed, just pinned to the non-stub version. When rdflib releases a new version with TS support included and you upgrade to that, at that point can the type definitions be removed. Apologies for the confusion.

Since three of five are fixed, and the two remaining are separate issues, can we close this issue?


Encode params for cleanse API the same way we do for other APIs

Code change for issue #10

Looks good @naikipl
Can you ship a new version? You'll need to update the version number and tweak the ChangeLog file


#145 Replace SipHasher with XxHash

Resolves #145
From an analysis by @Gankro it seems switching to xxhash is a sane choice for us and a reasonable default

Seems xxhash2 will be a better choose, which has streaming hasher and clearify API for xxhash32/xxhash64.
In fact, I'm working on a new PR to support LZ4 compression, which also need xxhash32 for block checksum and streaming hasher for content checksum.

@flier thanks for the feedback. in that case, it seems really like a better choice. i'm just about to leave my place now, but will return to this pr once i'm back again.

@flier i just had a quick look at xxhash2 and your lz4 PR. so far i understand, what twox-hash misses for you is the 32-bit variant, correct? if so, is there a chance for you to work with twox-hash'es author (ping @shepmaster) on a 32-bit variant? i'd prefer twox-hash due to its popularity, stability, and the fact that it's a pure-rust crate (looking at #149). having said that, i have no particular problem with xxhash2 and will prepare for its usage.

@xitep
Ok, I will evaluate the effort to implement those features in twox-hash.
Another difference is xxhash2 has a streaming hasher, which be used to generated content checksum for the LZ4 frame format, that is not required because it still TODO in Kafka 0.10.x.
For #149, I think the first step is to remove those 3rd party libraries, such as openssl, libsnappy etc, that could make our deployment more easy and smooth. After that we could replace remaining dependencies to the pure rust library, such as miniz-sys of flate2,  or xxhash2 etc.
What's you opinion?

@flier sounds good to me

@xitep I have submitted a PR (shepmaster/twox-hash#11) to upstream twox-hash

twox-hash will add 32-bit hash supports in shepmaster/twox-hash#12, we could wait them publish the new crate version, thanks @shepmaster.

:cool: :+1: many thanks! yes, let's wait for it.

I've just published version 1.1.0. Thanks so much for the interest!

@shepmaster thank you!


Bump version to 0.2, and upgrade libc to 0.2

Should be published after #326

Looks good - publish when you are ready (I can't - at work now)


dxForm colspan error

I'm trying layout content in 2 columns where the first column will be 3 times
If the item is not in the first column and the colspan is set, an error occurs:  E1025 - This layout is too complex to render
Template:
<dx-form colCount="4" [formData]="request" #form>
    <dxi-item colSpan="4" dataField="FirmId" [label]="{visible:false}">
        <bda-client-selector [(clientId)]="request.FirmId" [(readOnly)]="readOnly"></bda-client-selector>
        <!--<dxi-validation-rule type="required" message="Выберите клиента"></dxi-validation-rule>-->
    </dxi-item>
    <dxi-item [label]="{text:'Срок исполнения'}" dataField="DateEndByCreator" editorType="dxDateBox" [disabled]="readOnly">
        <dxi-validation-rule type="required"></dxi-validation-rule>
    </dxi-item>
    <dxi-item [label]="{visible:false}" dataField="Name" [editorOptions]="{placeholder:''}" colSpan="3" [disabled]="readOnly">
        <dxi-validation-rule type="required"></dxi-validation-rule>
    </dxi-item>
</dx-form>

The error appears inside the dxForm widget and seems not to be specific to the angular integration package. So it's better to create a ticket in DevExpress support center. Please attach a runnable project that shows markup of the bda-client-selector component and our support team will be happy to assist you.

@genusP I had the same issue before. Correction:
colSpan="3" => [colSpan]="3"

@kentsitu Thanks


Optional dependencies - Broadcaster Postgres dependency problem

Hello people!
First and foremost, I really appreciate your project! I am also using it heavily in a library I am making - eventhive.
So, I am trying to use your fastapi PubSub in a project that cannot use compiled C extensions.
This wouldn't be a problem as fastapi is Python, but broadcaster does require a postgres dependency that contains C code, failing my build.
I propose to create optional dependencies, exactly like broadcaster does, so I can exclude the postgres dependency that creates the issue - I'm not planning to use postgres anyway!
Thanks again for your project, it really is very useful!

HI @operatorequals :)
Thanks for reaching out and for the kind words :)
Eventhive sounds cool, and I'm happy our Pub/Sub impl. helps you there.
Sure, you can go ahead an open a PR to have that as part of the setup.py / requirements.txt ; please include a "default" or "all" option that will install all the dependencies as now, and one for each of the potential targets (e.g. postgres, redis, ... )
@roekatz or I will review and approve :)


General settings disappeared, scheduled tests stopped running

Describe the bug
Haven't logged into speedtest for a while and noticed no tests have been running for two months.  Also, general settings is missing in the web UI so there is no way to check the schedule.  Had been running fine for months.
Updated my image to 0.20.8 (from 0.20.6), no change.
To Reproduce
Unknown
Expected behavior
Speedtests should run per schedule, general settings should be available.
Environment (please complete the following information):

OS: Ubuntu / docker
Architecture: amd64
Browser: Firefox
Version: 0.20.8

Screenshots

Logs
error.log is empty
Additional context
N/A

same for me, i don't have settings general option.

Can you post your Compose-File?
Since 20.x the General-Settings Option is not available anymore, you have to define these in your Compose for example.
Please read the „Breaking Change“
Notice in the releases

speedtestn:
container_name: speedtestn
hostname: speedtestn
image: lscr.io/linuxserver/speedtest-tracker:latest
networks:
master_network:
ports:
- 50110:80
- 50113:443
environment:
- PUID=1000
- PGID=1000
- DB_CONNECTION=sqlite
- TZ=America/New_York
- PUBLIC_DASHBOARD=true
- APP_KEY=********************************* # How to generate an app key: https://speedtest-tracker.dev/
volumes:
- /docker_data/speedtestn/data:/config
- /docker_data/speedtestn/custom-ssl-keys:/config/keys
restart: unless-stopped

Is that contained like this in your Compose File?
As it says you have to generate an AppKey and replace the string after „APP_KEY=“ with that generated value.
Furthermore you can add the following to schedule a speedtest every 5 Minutes:
SPEEDTEST_SCHEDULE="*/5 * * * *"

thanks for the info.
I will add the SPEEDTEST_SCHEDULE
my app_key was masked with *** so it doesn't become public.

Thanks @Grimlogh for ref the releases, I'm going to close this out as OP looks all set.


Add a way to set ServerBuilderOptions on C++ interop server

****************************************************************

libgrpc.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]

****************************************************************

libgrpc++.so

     VM SIZE        FILE SIZE
 ++++++++++++++  ++++++++++++++

  [ = ]       0        0  [ = ]

[microbenchmarks] No significant performance differences

https://github.com/grpc/grpc/issues/15199


Fix gcsafe

Change was required to use HastyScribe in LiteStore

Actually this doesn't seem to hurt. I wonder why on Earth I used procs in the first place... but again, this was my first proper Nim project 😜
Accepting it, thanks!


Create Blog “2021-08-07-grafana-and-prometheus-logging”

Automatically generated by Netlify CMS

Unfinished blog entry, don't push this.


[Proofreading]: Harmentz - Blast Caliber tutorial needs line break

version: 0.0.235

The text for the Blast Caliber tutorial in Harmentz during the fight against the knights is slightly too long.

done


Fix Service filtering when clicking on service relation (#4517)

Pull request opened by the merge tool on behalf of #4517

Merged in d36c7f164b0701e334ec51dd69c67e40f58f08c6


#76 Added support for extension properties in external tasks

I have added support for extension properties for external tasks. This feature was introduced with camunda 7.14. It's required for our project.
https://docs.camunda.org/manual/7.14/reference/rest/external-task/fetch/#example-with-extension-properties

Hi @jlucansky,
Where do we go from here?
@EvilPingu

Hi Jan @jlucansky,
Do you plan to release this feature anytime soon?

Thanks @EvilPingu , I've also wanted to add support for this.
@jlucansky could you take a look at it? :)

Please have a look at #91


Bump kube versions in EKS-D releases list

Bumping up to corresponding Kubernetes versions in the releases file
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Ugh my bad


Question on hardware requirements for OPT-175B

Hello, and thank you for giving me access to OPT-175B! I'm working on setting it up on my institution's computing cluster. However, I noticed in the readme on the API functionality here, it says that the model can only be run on Azure, as it requires 80GB A100 GPUs.
On our cluster, I have access to nodes that each have 8 48GB A40 GPUs. Based on my calculations of the model size, I was thinking this should be sufficient to run the model. But does this seem feasible? I would try it myself directly first, but it takes a long time to acquire all these resources in the shared environment!
Any ideas or advice are appreciated. Thanks for your time!

@shanestorks  H!
We recently integrated OPT-175B serving with Alpa backend; it allows you to run big model training/inference on the cluster setup you described (lower-end GPUs than 80GB A100, but with sufficient total memory).
See this guide on how to set up!
I'm proposing a PR to put this integration in the Metaseq repo as well.

The MP16 work has landed (#214) for the faster 16x v100 support. Before that we did it with FSDP and MP8 but latency was AWFUL.

But total memory needs to exceed 350GB (so 8x48 is probably barely enough). For that you can just use the MP8 checkpoints.

Thank you all for the suggestions! These new developments give us options to make OPT work with our hardware. Will close this issue for now as we will try these suggestions.


[fb-package] Specify environment timezone

Description
Explicitly sets R environment timezone to LA/Pacific time.
Fixes
Fixes potential problem where we could get different output if we run the pipeline on a computer with a different system timezone. Based on some profiling, a small but not insignificant amount of run time is taken up by checking the computer's timezone setting, so this change stands to improve run time slightly.
Impact
Currently, the FB pipeline runs on a machine in Pittsburgh (Eastern time), so this change may impact indicator values. TBD: what is the magnitude of this change?

This approach does not have the expected behavior, that is, runs on a computer using Eastern time and on a computer using Eastern time but with environment timezone explicitly set to Pacific time produce the same output. It may be that the environment timezone is somehow not broad enough in scope.
Given a move towards running the survey pipeline and tables in a Docker container, in which we can set the container timezone explicitly, setting the R environment timezone seems unnecessary.


haproxy: 1.9.8 -> 2.0.8

Motivation for this change
New LTS release.
Also add support for the builtin prometheus exporter, so we don't have to use an external service.
The nixos test doesn't complete but currently on unstable all nixos tests are broken...
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

Notify maintainers
cc @fuzzy-id

@GrahamcOfBorg test haproxy

nix-review happy, macos 10.13


Small fixes related to .NET resource handling.

https://github.com/crackinglandia/pype32/issues/8#issuecomment-113225099

https://github.com/crackinglandia/pype32/issues/9#issuecomment-113255839

Squashed commits into one.

Added fixed mentioned in https://github.com/crackinglandia/pype32/issues/8#issuecomment-113902957

Thanks Sandor!


WIP: Implement vkGetRefreshCycleDurationGOOGLE() for macOS.

So far it always returned a 60 Hz refresh on macOS. Now we query the
actual refresh interval for the NSScreen on which our swapChain is likely
displaying.
Tested on a MacBookPro dual-display setup with builtin 60 Hz Retina panel
and external 100 Hz monitor, showing proper reporting depending on which
monitor the window was displaying on. Tested on macOS 10.15.7 only.

@kleinerm I have opened https://github.com/kleinerm/MoltenVK/pull/1 to implement the changes discussed in the review comments. It should allow us to work with AppKit as it doesn't use the delegate property, and it also uses the absolute top layer in case the layer we are presenting to is not the parent layer in the window.

Hi, sorry i don't have time to work on this further atm., and also I'm not a MoltenVK maintainer with any decision power. I think it would be better to direct your new pull request to the official MoltenVK repo here, so the maintainers can check if this is the way forward, maybe referencing this pull request for context.

Closing. Superseded by #1596.