The Nadam optimizer is non-deterministic due to a race condition related to inter_op_parallelism_threads

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes.
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Arch Linux
TensorFlow installed from (source or binary): Arch Linux repositories
TensorFlow version (use command below): 'unknown' 1.13.1
Python version: 3.7.3
GPU model and memory: None. I'm running on CPU.

Describe the current behavior
When inter_op_parallelism_threads is set to 1 the iterations variable in Nadam optimizer from tensorflow/python/keras/optimizers.py always starts at 1 when the optimizer is executed; when inter_op_parallelism_threads is higher than 1 then iterations sometimes starts as 0. The higher the inter_op_parallelism_threads is set to the higher the chance that iterations will start at 0 instead of at 1.
(I know this because I tf.Print'd everything inside of the Nadam optimizer.)
This makes the training results non-deterministic since the Nadam optimizer depends on the current iteration number when calculating the weight updates.
Describe the expected behavior
The Nadam optimizer will be deterministic.
Code to reproduce the issue
#!/usr/bin/python

import tensorflow as tf;
import numpy as np;

constant = tf.keras.initializers.constant;

inter_op_parallelism_threads = 1
# inter_op_parallelism_threads = 20

session_conf = tf.ConfigProto(intra_op_parallelism_threads=1, inter_op_parallelism_threads=inter_op_parallelism_threads)
graph = tf.get_default_graph()
session = tf.Session(graph=graph, config=session_conf)
tf.keras.backend.set_session(session)

optimizer = tf.keras.optimizers.Nadam(lr=1.0, beta_1=0.9, beta_2=0.999, epsilon=1e-7)

bias = constant(np.array([1.0]))
weights = constant(np.array([1.0]))
layer = tf.keras.layers.Dense(1, bias_initializer=bias, kernel_initializer=weights, input_shape=(1,))

model = tf.keras.Sequential()
model.add(layer)
model.compile(optimizer=optimizer, metrics=["accuracy"], loss=["mean_squared_error"])

input = np.array([1.0])
output = np.array([1.0])
model.train_on_batch(input, output)

w = layer.get_weights()
print(w)

Output when inter_op_parallelism_threads is set to 20 (so iterations starts at 0):
[array([[-0.0564518]], dtype=float32), array([-0.0564518], dtype=float32)]

Output when inter_op_parallelism_threads is set to 1 (so iterations starts at 1):
[array([[-0.49368942]], dtype=float32), array([-0.49368942], dtype=float32)]

@koute : I have tried to reproduce the issue on colab (tf1.13.1) but unfortunately I am able to get the same result in both the cases mentioned (inter_op_parallelism_threads = 1 or inter_op_parallelism_threads = 20). Please let us know if it is still an issue. Thanks!

@achandraa Yes. I have no idea why it doesn't reproduce on colab; it does locally on my PC every time I run it. It's a race condition though, so I'm not surprised.
It does reproduce in a different way on colab though. Select "GPU" in "Runtime" -> "Change runtime type" and you'll always hit the case where iterations starts at 0 (so you'll get -0.05648327; it's not exactly the same as in my original reproduction most likely due to the fact that now it's running on the GPU instead of the CPU); if you select "None" then you'll always hit the case when iterations starts at 1 (so you'll get -0.49368942 as result). I believe the root cause of this should be the same as my original issue.

@koute : Tried in Colab (GPU runtime) as suggested but got the same result for cases where inter_op_parallelism_threads = 0 or inter_op_parallelism_threads = None which is [array([[-0.05648327]], dtype=float32), array([-0.05648327], dtype=float32)]. And this is on tensorflow version 1.13.1. Can you try and let us know whether you are getting different on Colab with same environment. Thanks!

@achandraa Yes, you're getting the same results as I do. Now change the runtime to CPU, and you'll always get -0.49368942.
I'll reiterate what I said - on Colab the original issue does not reproduce in the way as originally described, i.e. the parallelism settings do not matter, but it does reproduce in a different way. Here's a handy table:
| Runtime | initial value of iterations | Result      |
| --------------------------------------------------- |
| CPU     | 1                           | -0.49368942 |
| GPU     | 0                           | -0.05648327 |
| --------------------------------------------------- |

The results on both runtimes should be the same. I believe the root cause of this problem should be the same as the root cause of my original problem.

I got following output while running on different runtimes on Colab with version 1.13.1 :
| Runtime | initial value of iterations | Result        |
| ------------------------------------------------------------   |
| CPU       | 1                                 | -0.0564518   |
| GPU       | 0                                 | -0.05648327 |
| ------------------------------------------------------------   |
This is what we are referring to ?

So you're getting the same values on both the CPU and the GPU runtime? This is really strange. Because that's not what I am getting. Are you perhaps getting assigned a slightly different environment on Colab or something like that? (I'm not really too familiar with how exactly it works.)
Here are the screenshots of my results:
CPU results - https://imgur.com/eqhsNTn
GPU results - https://imgur.com/dj2uxNg
This is on the default Colab environment in a fresh Python 3 notebook.

After checking again and trying on default Colab environment in a fresh Python 3 notebook with TensorFlow version 1.13.1,  I was able to reproduce the issue with the outputs you have provided. Below  are the result :
Runtime : CPU
Initial value of iterations : 1
Result : -0.49368942
Runtime : GPU
Initial value of iterations : 0
Result : -0.05648327

@koute
Please post this issue on keras-team/keras repo.
To know more see;
https://discuss.tensorflow.org/t/keras-project-moved-to-new-repository-in-https-github-com-keras-team-keras/1999


Reverse SERVO direction

I am trying to get joy_teleop working. What will fix it if servo (steering) direction is reversed (left steers to the right and right joy steers it to the left) ?. I am trying to find that point in the code where I can reverse the command. Any help is appreciated.

You can try to negate the scale on this line here.
If that doesn't work, try to adjust the values on lines 16 and 17 here to reverse the direction.

Your first suggestion worked, thanks Billy . Launching joy_teleop, I can control the steering and motor for just a few seconds and after that non of joy control commands work. At the same time, I can see the following logs in terminal:
[vesc/low_level/ackermann_cmd_mux-10] process has finished cleanly
[vesc/high_level/ackermann_cmd_mux-8] process has finished cleanly

Node remains running without further error. There is just lack of control very soon after launching the node. Should I open a new issue for this ?

Are there any other error messages?
Can you provide your exact VESC hardware and firmware version, and any modifications you made to the packages?

There is no any other error. I am using VESC hw 4.2 and fw 1.8. The VESC pkg I am using is the one provided by MIT team and is compatible with hw 4.2. The only change I made to the package is that the joystick  (joy pkg) runs on different machine and the joy_teleop on the car subscribe to /joy over network. I cannot see any network delay here, car receives joy commands almost in real-time.

Ok, your setup is uncharted territory for us so unfortunately, I might not be very helpful.
From some searching online it sounds like it could be a system time issue between the remote and the car. The command mux nodelet might be throwing errors because the stamped time from the joy messages is ahead of the system time on the car.
Other than that I have no idea what the error could be.


new

@GUDAPATIVENKATESH Please sign the Contributor License Agreement!
Click here to manually synchronize the status of this Pull Request.
See the FAQ for frequently asked questions.

done


Make Foundry CI output tests easier to read on failure

Make tests with Foundry have the same structure as the ones we used to have for truffle.

@soloseng great catch!


5.0 Update routing comments about default catch-all behavior

We have a disclaimer indicating an existing issue with regards to routing in 3.1 on our docs. (https://github.com/aspnet/aspnetcore/issues/18677). We made this behavior the default in 5.0, so we should update the docs accordingly.

cc @serpent5

@javiercn  next time,  Can you create an issue from the doc page? That will connect the doc to the issue. Go to the bottom of the page and select This page .


Allow intermixing shortcut- and key-remapping

Enable mapping a shortcut to a single key (and vice versa)
The expected behaviour would be to set the following mappings:
Alt + o              ->     ö
Shift + Alt + o   ->    Ö
What works already is the second example, since you can make it to produce Shift + ö (which then prints as Ö).
What doesn't work is the first example, since key remapping does not allow for using shortcut sequences (instead only single keys) and on the other hand shortcut remapping does not allow for a single keystroke be its result ("Shortcut must start with a modifier key" when trying to set ö as the result of Alt + o shortcut).
For me the scenario would be to get rid of German QWERTZ keyboards and be able to do programming with QWERTY, while also being able to type proper German text with Umlauts. All of that while not having to use a crude keyboard layout where, e.g.:
RAlt + q = ä
RAlt + p = ö
RAlt + y = ü
Maybe the way to go would be to either add another category that allows intermixed (shortcut -> single keystroke) mappings, or just allow exactly that for the existing shortcut remapping category.

If you'd like to see this feature implemented, add a 👍 reaction to this post.

just for the record: my plan doesn't work, since the remapping is a direct key remapping and not putting in chars.
So if I press the aforementioned combination of keys, while the keyboard is set to US QWERTY, it just prints whatever character is on that position on the QWERTY layout and not the QWERTZ layout.


docs(readme/quick-start): remove "global install" section, it's not r…

Webpack, rimraf, and webpack-dev-server are all already included in package.json as devDependencies.
This means they'll be in node_modules/.bin.
npm scripts implicitly have node_modules/.bin in their path, so the npm scripts will then read from node_modules/.bin, making the global install extraneous.

while true. without this I'll get 20 issues from windows users


Playback speed change?

Just out of curiosity, is playback speed change possible (playback in certain bpm for practice etc.)

I can't see anything like that in the Spotify API So I would guess the answer is no.

Yup, another solution would be to buffer enough content, use a GStreamer backend and have a custom GStreamer pipeline etc... but that's definitely beyond my abilities! Marking this as wontfix for now


Add auto complete

Uses argcomplete to provide autocompetion for argparse.
Adds checks to make sure -s and -d csv lists are valid.
Adds this to dependencies.sh to enable argcomplete in bash: sudo activate-global-python-argcomplete

Needs rebase


Filter should be cleared on Pressing Escape even when tree elements are focused

Testing #54935

Open breadcrumb tree
Start typing to filter
An input box opens and text is entered there and tree start gets filtering
Press down arrow to focus filtered tree elements
Now press escape

It should clear the filter

Closing as-designed because this is inline with quick pick


Performance issue

Hi,
I tested the ParquetSource on a parquet file of about 5M records, split in 144 partitions (files) on an 8 CPU server in the default way:
    ParquetSource(new Path(modelPath))
      .toFrame(16)
      .toList
      .map(convertRow)

After about 10 min the process got stuck and I had to kill it. All cpu's were at 100% for the whole duration
I then tested with a slightly modified codebase:
    ParquetSource(new Path(modelPath))
      .parts
      .par
      .flatMap { part =>
        part.data
          .map(convertRow)
          .toList.toBlocking.single
      }

And that finished the job in 30 sec. Even without the .par line it still only takes 2 min (running on a single cpu)
Any idea why the two implementations give so different results?

Possibly some kind of infinite loop error. I'll look into it.
On 24 Nov 2016 9:55 a.m., "Niek Bartholomeus"<EMAIL_ADDRESS>wrote:

Hi,
I tested the ParquetSource on a parquet file of about 5M records, split in
144 partitions (files) on an 8 CPU server in the default way:
ParquetSource(new Path(modelPath))
  .toFrame(16)
  .toList
  .map(convertRow)

After about 10 min the process got stuck and I had to kill it. All cpu's
were at 100% for the whole duration
I then tested with a slightly modified codebase:
ParquetSource(new Path(modelPath))
  .parts
  .par
  .flatMap { part =>
    part.data
      .map(convertRow)
      .toList.toBlocking.single
  }

And that finished the job in 30 sec. Even without the .par line it still
only takes 2 min (running on a single cpu)
Any idea why the two implementations give so different results?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/sksamuel/eel-sdk/issues/180, or mute the thread
https://github.com/notifications/unsubscribe-auth/AAtZGsc-pG6XomAPu_Erw6qJpgVp8UyFks5rBV71gaJpZM4K7dpH
.

Do you have a sample file? If not, can you tell me how unique the data is (how many values are repeated roughly)

The input file contains words with an array of doubles:
  private def convertRow(row: Row) = {
    val word = row.values(0).asInstanceOf[String]
    val vector = row.values(1).asInstanceOf[util.ArrayList[Double]].toArray.map(_.asInstanceOf[Record].get(0).asInstanceOf[Double])
    (word, new DenseVector(vector))
  }

All words are unique.
Let me know if there's anything else I can do to help. Thanks!

After upgrading to 1.1.0-snapshot I had to slightly modify my workaround solution as the part was changed from Observable to Flux:
    ParquetSource(new Path(modelPath))
      .parts
      .par
      .flatMap { part =>
        logger.info(s"Loading part...")
        part.data
          .toIterable.asScala
          .map(convertRow)
      }

But that gives a huge performance hit: the 5 million records are loaded now in 1m40s min compared to 30s before. Just wanted to let you know.

That's interesting. I think the .toIterable is blocking and slow. I certainly won't publish anything in 1.1.0 that's not up to scratch. I've been playing about with https://projectreactor.io/ as an alternative to RxJava1 that I was using before.

@tolomaus so I know the reason for the slowness now, the reactor-core and rxjava frameworks use recursion when merging together multiple observables. If you have 100 files say, you have 100 levels of recursion for every single row. I'll be switching that out this week.

Good spot! As soon as this change is made I will switch back to the latest version on github and try it out.

Want to try that latest snapshot see what your results are?

Looks good!
I currently don't have the full dataset of 5M words available but I tested with roughly a fifth of it. With the previous version the load took 15s and with the latest version it only takes 10s!
I will let you know the figures once I have my 5M dataset back.
One note: as I want to run parallel calculations based on the loaded dataset, I prefer to keep it split up in a ParMap. The parts api allows me to do this so I'm not using the ``toFrame``` api. I noticed however that the latest version is less straightforward to collect all rows per part:
    ParquetSource(new Path(wordVariationPath))
      .parts
      .par
      .flatMap { part =>
        var rowsTotal = List[Row]()
        part.iterator.foreach { rows =>
          rowsTotal = rowsTotal ++ rows
        }
      }

Is this the best way to proceed?
Thanks!

Yes you can do that, or the Frame API will allow you to multi-thread the
code in the next release as well. The reason for the List[Row] is that it
avoids a lot of the contention on the locks.
On 7 December 2016 at 09:58, Niek Bartholomeus<EMAIL_ADDRESS>wrote:

Looks good!
I currently don't have the full dataset of 5M words available but I tested
with roughly a fifth of it. With the previous version the load took 15s and
with the latest version it only takes 10s!
I will let you know the figures once I have my 5M dataset back.
One note: as I want to run parallel calculations based on the loaded
dataset, I prefer to keep it split up in a ParMap. The parts api allows
me to do this so I'm not using the ``toFrame``` api. I noticed however that
the latest version is less straightforward to collect all rows per part:
ParquetSource(new Path(wordVariationPath))
  .parts
  .par
  .flatMap { part =>
    var rowsTotal = List[Row]()
    part.iterator.foreach { rows =>
      rowsTotal = rowsTotal ++ rows
    }
  }

Is this the best way to proceed?
Thanks!
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
https://github.com/sksamuel/eel-sdk/issues/180#issuecomment-265404885,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAtZGgT7C9eyM2ppEvlit6F2mdun5rvYks5rFoNBgaJpZM4K7dpH
.

I've released another snapshot. There's no more reactive framework slowing it down in the middle. Also, the ParquetSource is now a native parquet source (no avro middle man), and the newly added AvroParquetSource is what was called ParquetSource before. The new one is not well tested yet, and the type mappings might still need correction. @hannesmiller @tolomaus

My version of parquet somehow got bumped from 1.8.1 to 1.9.0 today which threw an error while parsing the data:
Caused by: java.lang.ClassCastException: org.apache.parquet.example.data.simple.SimpleGroup cannot be cast to org.apache.parquet.example.data.simple.BinaryValue
	at org.apache.parquet.example.data.simple.SimpleGroup.getString(SimpleGroup.java:124)
	at io.eels.component.parquet.ParquetDeserializer$$anonfun$toRow$1.apply(ParquetDeserializer.scala:47)
	at io.eels.component.parquet.ParquetDeserializer$$anonfun$toRow$1.apply(ParquetDeserializer.scala:34)
	at scala.collection.immutable.Range.foreach(Range.scala:160)
	at io.eels.component.parquet.ParquetDeserializer.toRow(ParquetDeserializer.scala:34)
	at io.eels.component.parquet.ParquetDeserializer$$anonfun$toRow$1.apply(ParquetDeserializer.scala:46)
	at io.eels.component.parquet.ParquetDeserializer$$anonfun$toRow$1.apply(ParquetDeserializer.scala:34)
	at scala.collection.immutable.Range.foreach(Range.scala:160)
	at io.eels.component.parquet.ParquetDeserializer.toRow(ParquetDeserializer.scala:34)
	at io.eels.component.parquet.ParquetPart$$anon$1$$anonfun$1$$anonfun$apply$1.apply(ParquetPart.scala:21)

I was still using the initial 1.1.0 snapshot and I'm not sure what triggered this bump, nor how I can revert it to the working situation. So I'm back now to eel version 1.0.1 until I have a bit more time to investigate this issue.
The parquet file was generated by spark which is still using parquet 1.7.0, maybe there are backward compatibility issues between 1.7.0 and 1.9.0.

@tolomaus Hope you had a nice Christmas.
Release 1.1.0-M2 has a new native parquet reader/writer, which skips avro and goes directly from parquet to eel row for extra performance. @hannesmiller and I have also spent a lot of time going over the parquet spec to ensure we're compliant, and I've added spark compatibility tests also. Give it a go and see how you get on - if there's any niggles with type conversions (or anything else) paste them up.

I should add, if you want to use the original avro based parquet source and sink, you can, AvroParquetSource(path) and AvroParquetSink(path). The new ones are the names as before, ParquetSource and ParquetSink. The issues you were seeing between 1.7 and 1.9 parquet I actually think was caused by the fact the latest snapshot had a half finished version of the new native reader.

Thanks merry Christmas and happy new year for you and the awesome eel team as well! ;-)
I upgraded spark from 2.0.2 to 2.1.0 yesterday and noticed they have also bumped their parquet dependency from 1.7 to 1.8.1. But I guess the parquet versioning may not be the root cause if I used a half finished reader.
Anyway, I will upgrade my eel dependency to 1.1.0-M2 and see how it goes.
(btw I see quite some people starting to use spark job server to access their offline generated datasets in an online - think web server - context. Imo eel is a much better, more lightweight solution. Do you also see these two approaches as alternatives?)

Hi Niek,
For our initial Ingestion requirements into our Hadoop cluster I am not sure what sure what added value the Spark Job Server is going to give us on the project Sam and I working on?
We ingest flat CSV files or Oracle datasets coming from upstream systems.
Having said that we extensively use  Spark for parallel processing the datasets already resident in HDFS.
The datasets got their in the first place via EEL however we do create materialised views (or derived datasets) based on the original data - for this we simply use the Spark save API.
We launch individual Spark jobs for our calculation and reporting requirements - we create a Spark context for each job.
The advantage for us is we can specifically tune each job for the required resources and relinquish them when the job has finished - which keeps everyone happy in a multi-tenant cluster environment.
The disadvantage I guess is the initial start up time for each discrete spark job - if you have a long running server with all the resources pre-allocated then each job is going to be quicker. In addition you can cache your Spark Datasets.
But how much of the resources (CPU, RAM) are you going to allocate? Remember long running processes means you take them away from others.
I believe Spark Job Server is a long running process with pre-allocated resources?
However the Spark Job Server looks interesting because it looks lightweight due to its Rest interface and the fact you can cache and share DataFrames/Datasets for multiple users.
We are looking at Apache Ignite caching to achieve this for our own needs.
Regards,
Hannes

Hi Hannes,
Thanks for your explanations. I was specifically referring to using the spark job server as a "Spark as a Query Engine" i.e. loading a parquet file in memory and querying it in multi user mode, compared to simply using eel for this.
I don't see any value in using spark job server either, though I see some people doing it e.g. https://aws.amazon.com/blogs/big-data/installing-and-running-jobserver-for-apache-spark-on-amazon-emr/

I switched to 1.1.0-M2 but ran into an error when loading the parquet:
Execution exception[[ParquetDecodingException: Can not read value at 0 in block -1 in file file:/data/parquet/en/W2vWordVectors/part-r-00060-a49d7a26-5434-4314-9c15-295089627380.snappy.parquet]]
	at play.api.http.HttpErrorHandlerExceptions$.throwableToUsefulException(HttpErrorHandler.scala:293)
	at play.api.http.DefaultHttpErrorHandler.onServerError(HttpErrorHandler.scala:220)
	at play.api.GlobalSettings$class.onError(GlobalSettings.scala:160)
	at play.api.DefaultGlobal$.onError(GlobalSettings.scala:188)
	at play.api.http.GlobalSettingsHttpErrorHandler.onServerError(HttpErrorHandler.scala:100)
	at play.core.server.netty.PlayRequestHandler$$anonfun$2$$anonfun$apply$1.applyOrElse(PlayRequestHandler.scala:100)
	at play.core.server.netty.PlayRequestHandler$$anonfun$2$$anonfun$apply$1.applyOrElse(PlayRequestHandler.scala:99)
	at scala.concurrent.Future$$anonfun$recoverWith$1.apply(Future.scala:346)
	at scala.concurrent.Future$$anonfun$recoverWith$1.apply(Future.scala:345)
	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
Caused by: org.apache.parquet.io.ParquetDecodingException: Can not read value at 0 in block -1 in file file:/data/parquet/en/W2vWordVectors/part-r-00060-a49d7a26-5434-4314-9c15-295089627380.snappy.parquet
	at org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:223)
	at org.apache.parquet.hadoop.ParquetReader.read(ParquetReader.java:122)
	at org.apache.parquet.hadoop.ParquetReader.read(ParquetReader.java:126)
	at io.eels.component.parquet.util.ParquetIterator$$anonfun$apply$1.apply(ParquetIterator.scala:10)
	at scala.collection.Iterator$$anon$9.next(Iterator.scala:162)
	at scala.collection.Iterator$$anon$16.hasNext(Iterator.scala:598)
	at scala.collection.Iterator$GroupedIterator.fill(Iterator.scala:1126)
	at scala.collection.Iterator$GroupedIterator.hasNext(Iterator.scala:1132)
	at io.eels.CloseableIterator$$anon$1.hasNext(CloseableIterator.scala:10)
	at scala.collection.Iterator$class.foreach(Iterator.scala:893)
Caused by: java.lang.RuntimeException: Unsupported type StructType(Vector(Field(list,StructType(Vector(Field(element,io.eels.schema.DoubleType$@70c66b76,false,false,None,Map()))),false,false,None,Map())))
	at scala.sys.package$.error(package.scala:27)
	at io.eels.component.parquet.RowRecordMaterializer$$anon$1.getConverter(RowReadSupport.scala:64)
	at org.apache.parquet.io.RecordReaderImplementation.<init>(RecordReaderImplementation.java:266)
	at org.apache.parquet.io.MessageColumnIO$1.visit(MessageColumnIO.java:140)
	at org.apache.parquet.io.MessageColumnIO$1.visit(MessageColumnIO.java:106)
	at org.apache.parquet.filter2.compat.FilterCompat$NoOpFilter.accept(FilterCompat.java:154)
	at org.apache.parquet.io.MessageColumnIO.getRecordReader(MessageColumnIO.java:106)
	at org.apache.parquet.hadoop.InternalParquetRecordReader.checkRead(InternalParquetRecordReader.java:136)
	at org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:194)
	at org.apache.parquet.hadoop.ParquetReader.read(ParquetReader.java:122)

Here is my code:
ParquetSource(new Path(wordVectorPath))
      .parts
      .par
      .flatMap { part =>
        var rowsTotal = List[Row]()
        part.iterator.foreach { rows =>
          rowsTotal = rowsTotal ++ rows
        }
        rowsTotal.map(convertWordVectorRow)
      }

  private def convertWordVectorRow(row: Row) = {
    val word = row.values(0).asInstanceOf[String]
    val vector = row.values(1).asInstanceOf[util.ArrayList[Double]].toArray.map(_.asInstanceOf[Record].get(0).asInstanceOf[Double])
    val label = row.values(2).asInstanceOf[String]
    val existsAsRealWordInVocabulary = row.values(3).asInstanceOf[Boolean]
    ...
}

The parquet file is still v1.7 but I will rerun the model with spark 2.1 and try again with the regenerated parquet 1.8.1 file, just to be sure.

I see...we use out Apache Zeppelin which has a nice SQL query interpreter for Hive.

The Scala interactive interpreter allows you load the Parquet Files, Register a temp table and perform SQL on it the temp table

This is the error affecting @tolomaus
java.lang.RuntimeException: Unsupported type StructType(Vector(Field(list,StructType(Vector(Field(element,io.eels.schema.DoubleType$@70c66b76,false,false,None,Map()))),false,false,None,Map())))
	at scala.sys.package$.error(package.scala:27)
	at io.eels.component.parquet.RowRecordMaterializer$$anon$1.getConverter(RowReadSupport.scala:64)

BTW orc.apache.org support is now slightly faster than Parquet. Seems they're doing a good job on it and doing regular releases.

Hi @tolomaus. Thanks for your continued testing. Do you want to give M3 a quick go?

Or even better, try 1.1.0 final.

I slightly modified the line that converts the vector, an Array of Doubles:
val vector = row.values(1).asInstanceOf[Vector[Double]].toArray

but I get an error:
java.lang.ClassCastException: scala.collection.immutable.Vector cannot be cast to java.lang.Double

Looking at it in debug mode I see that something is wrong with the structure of the data: the first item in the vector is correctly defined as a double, but the rest are vector(double)'s instead of simply doubles
E.g.:
(Vector(-0.089236, Vector(-0.08471), Vector(-0.05365), Vector(-0.030827), ...

Is this an existing file you're reading in, or a new file you're writing out? Because I can't see immediately why this would break, but saying that my tests are only checking the string case.
https://github.com/sksamuel/eel-sdk/blob/master/eel-components/src/test/scala/io/eels/component/parquet/ParquetComponentTest.scala#L78-L103
I'll add another test for doubles and post back.

Ok test added.
https://github.com/sksamuel/eel-sdk/blob/master/eel-components/src/test/scala/io/eels/component/parquet/ParquetComponentTest.scala#L104-L129
If the data was written out with an earlier snapshot version of eel, I'd re-write it to be sure. If it was created outside of eel then it must be a bug and maybe its in processing the nested schema. Are you able to paste up your schema, obfuscate the names if you have to, its just the structure that's important.

I'm only reading files. I double checked the file contents using spark and it looks fine from there:
+-----------+--------------------+
|       word|              vector|
+-----------+--------------------+
|clienthandl|[-0.079246, 0.050...|
+-----------+--------------------+

Here is the eel schema:
StructType(Vector(Field(word,io.eels.schema.StringType$@45045889,true,false,None,Map()), Field(vector,StructType(Vector(Field(list,ArrayType(StructType(Vector(Field(element,io.eels.schema.DoubleType$@5ee7b56d,false,false,None,Map())))),true,false,None,Map()))),true,false,None,Map()), Field(label,io.eels.schema.StringType$@45045889,true,false,None,Map()), Field(existsAsRealWordInVocabulary,io.eels.schema.BooleanType$@67ac493c,true,false,None,Map())))

See here the schema from spark:
StructType(StructField(word,StringType,true), StructField(vector,ArrayType(DoubleType,true),true), StructField(label,StringType,true), StructField(existsAsRealWordInVocabulary,BooleanType,true))

Thanks I'll create a file that mimivs this schema in spark and then try
reading. I suspect the error is in the code that combines values to form
vectors.
On 12 Jan 2017 7:50 p.m., "Niek Bartholomeus"<EMAIL_ADDRESS>wrote:

See here the schema from spark:
StructType(StructField(word,StringType,true),
StructField(vector,ArrayType(DoubleType,true),true),
StructField(label,StringType,true), StructField(
existsAsRealWordInVocabulary,BooleanType,true))
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
https://github.com/sksamuel/eel-sdk/issues/180#issuecomment-272263956,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAtZGpoOpNABUyr-DGn8I1t-cHSOrFhoks5rRoP5gaJpZM4K7dpH
.

I got it to work by removing the head:
val vector = row.values(1).asInstanceOf[Vector[Any]].tail.map(_.asInstanceOf[Double]).toArray

So it looks like there first item from the vector should also be a Vector(double), just like the other items

Ahh that makes more sense. Hows the performance?
On 12 Jan 2017 8:05 p.m., "Niek Bartholomeus"<EMAIL_ADDRESS>wrote:

I got it to work by removing the head:
val vector = row.values(1).asInstanceOf[Vector[Any]].tail.map(_.asInstanceOf[Double]).toArray
So it looks like there first item from the vector should also be a
Vector(double), just like the other items
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
https://github.com/sksamuel/eel-sdk/issues/180#issuecomment-272268402,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAtZGkJRn2DCB-0wYFOzp5XWkMYfplzWks5rRoeRgaJpZM4K7dpH
.

Slightly slower: 13s compared to 16s in 1.0.1. This is for about 800000 items. Not what we expected :-) 1.1.0 uses a more "native" api right? Strange...

Slower but 13 is quicker than 16 lol
On 12 Jan 2017 8:17 p.m., "Niek Bartholomeus"<EMAIL_ADDRESS>wrote:

Slightly slower: 13s compared to 16s in 1.0.1. This is for about 800000
items. Not what we expected :-) 1.1.0 uses a more "native" api right?
Strange...
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
https://github.com/sksamuel/eel-sdk/issues/180#issuecomment-272271504,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAtZGqJ9F2qHHKDhsFA4orc1nb5YJP3Yks5rRopSgaJpZM4K7dpH
.

see my edited post ;-)

Do you want to try the AvroParquetSource

Sure, I switched to AvroParquetSource, but it looks like it uses the ArrayList from 1.0.1:
Caused by: java.lang.ClassCastException: java.util.ArrayList cannot be cast to scala.collection.immutable.Vector

I will see if I can fix it tomorrow and get back with the results

Ahh yeah course.
On 12 Jan 2017 8:31 p.m., "Niek Bartholomeus"<EMAIL_ADDRESS>wrote:

Sure, I switched to AvroParquetSource, but it looks like it uses the
ArrayList from 1.0.1:
Caused by: java.lang.ClassCastException: java.util.ArrayList cannot be cast to scala.collection.immutable.Vector
I will see if I can fix it tomorrow and get back with the results
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
https://github.com/sksamuel/eel-sdk/issues/180#issuecomment-272274993,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAtZGv4djHFx8RIyqUucP8SZ1VT6dCLDks5rRo2mgaJpZM4K7dpH
.

Should cast to a Seq really because no guarantee you'll get a Vector out of the sources. I tend to return those, but if the underlying library returns an Array I reuse that rather than convert to a Scala col.

Ok thanks. Would be good if we could map the source format directly to a case class and let the library take care of the conversion ceremony.
Is that something you could add to this library as well? Maybe through avro4s? Or as a quick alternative expose a GenericRecord so I can pass that myself to avro4s?
val ennio = format.from(record)

Yes that's a good idea and one touched upon before. I'll be adding that for 1.2.
I'm about to release 1.1.1 which has the regression on speed fixed, and AvroParquetSource struct bug fixed. I think this time should be faster than the previous best.
Marshalling to case classes will slow things down a lot though, expect throughput to halve at least, from the raw Seq[Any] case at least.

See Type safe Frames #140

Ok 1.1.1 has been released to central

Hi Sam,
Do the performances enhancements  on AvroParquet also apply to native Parquet?
I have some performance issues related with JdbcSource -> HiveSink with a Parquet dialect - I just want to establish this before embarking on the root cause of my import performance issues.
Cheers,
Hannes

The hive sink uses native parquet but the regression and fix only apply to
reading parquet.
On 13 Jan 2017 09:29, "hannesmiller"<EMAIL_ADDRESS>wrote:
Hi Sam,
Do the performances enhancements on AvroParquet also apply to native
Parquet?
I have some performance issues related with JdbcSource -> HiveSink with a
Parquet dialect - I just want to establish this before embarking on the
root cause of my import performance issues.
Cheers,
Hannes
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
https://github.com/sksamuel/eel-sdk/issues/180#issuecomment-272398771,
or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAtZGteDEFLMEGnl-kq2YyIRC37EokQcks5rR0PdgaJpZM4K7dpH
.

@tolomaus Did you manage to try 1.1.1 because that should be the fastest of them all.

Ah great the conversion issue seems to be fixed!
I compared the performance of 1.1.1 with 1.0.1 but it's still a bit slower: 13s vs 11s (this is a different dataset so the results are not comparable to the previous figures)

I had a closer look an it turns out that the conversion issue is still there, I didn't notice initially because I still had the ".tail" in my code which drops the badly converted head of the list.
After running it a couple of times, I can say that the performance is quite on par between both versions.

I'll close this issue then if the performance is now back up. If you were able to open up a new ticket for the conversion error with a sample as I cannot seem to duplicate that would be good. If not I'll keep trying.

Yes indeed, the performance is fine now. Ok I will see if I can reproduce the conversion error with a minimal scenario.


.begin() always returns 255

Hi Tedy - thanks for writing this lib.
I think I've found a bug:
During a .begin, a reset() is issued. Immediately after, setGain() is issued. setGain fails ~80% of the time (returns 255) because the lib immediately tries to read the first memory location
uint8_t status1 = readRegister(GAIN_SEL_REG, old_val);

and it can't because the IC is still resetting. This causes the .begin to return 255 (error).
POR is 1.6ms max. Software reset time limit is not specified.  We could poll register until it reads correctly, but I did a bit of a hack in the reset() function:
uint8_t
MLX90393::
reset()
{
  invalidateCache();
  uint8_t cmd = CMD_RESET;

  sendCommand(cmd); //Device now resets. We must give it time to complete
  delay(2); //POR is 1.6ms max. Software reset time limit is not specified. 2ms was found to be good.
  return sendCommand(cmd);
}

This works well for me. I'll try to issue a pull request in a moment.

Yes, my pleasure.  Thanks for your interest and enhancements, and, you know ... Sparkfun.
I commented in the merge about either exposing the delay as a defaulted parameter or doing polling as you suggest, but for the time being, the 2ms delay seems fine.

SparkFun = no problem :) Thanks!
Passing delay by argument with default is a good solution. I'll try to do that the next time I'm in the lib.


⚠ PrivateHD has degraded performance

In 6160bc6, PrivateHD ($PHD) experienced degraded performance:

HTTP code: 200
Response time: 5094 ms

Resolved: PrivateHD performance has improved in 41fe59e.


docs(Header): fix typo in component example description

As an alternative to merging this PR as-is, you may want to remove the word "an" altogether (and not replace it with anything).

Codecov Report

Merging #2340 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #2340   +/-   ##
=======================================
  Coverage   99.73%   99.73%
=======================================
  Files         152      152
  Lines        2655     2655
=======================================
  Hits         2648     2648
  Misses          7        7

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 003fd19...2efc91b. Read the comment docs.

@levithomason My English is poor, please take a look there. Is suggested fix correct?

This is an improvement, thanks.

Released in<EMAIL_ADDRESS>


SIGSEGV: Illegal storage access. (Attempt to read from nil?)

Amazing progress on this!
After pulling latest changes from the last 2 days, I now get this error when attempting to run the basic_controls example program:
$ ./basic_controls
Traceback (most recent call last)
basic_controls.nim(4)    basic_controls
uibuilder.nim(293)       load
uibuilder.nim(142)       parseXml
uibuilder.nim(142)       parseXml
uibuilder.nim(142)       parseXml
uibuilder.nim(142)       parseXml
uibuilder.nim(125)       parseXml
system.nim(1529)         add
SIGSEGV: Illegal storage access. (Attempt to read from nil?)

I think you need Nim devel version


Possibility of zooming on pictures

I have just added, with 426a0a8c4103edfa99d3b15167241a700dcb5c5b and de73c069dd17ca36831973dfc0ac071c95d03fe6, visual representations that I hope will ease the understanding of the differences and relationships between languages (cf. just above https://csci-1301.github.io/book.html#software-concepts-1 ).
However, the pictures are not really visible (especially the second), and it would be beneficial to have a way of zooming on the pictures in the html version of the book. A jquerry zoom plug-in could maybe implement this feature?

Of course, there is always the option of telling the student to do a simple right click -> view image, if we don't want to go that road…

I like this idea but can we find something similar that does not depend on jquery, ideally no dependencies?

Around here they suggest to simply add a link to the original image from the image, but that creates all kinds of complications (the "auto-completion" of the file path with svg or pdf based on the output cannot work anymore, the figure environment is butchered…).
If we want to avoid dependencies (which I think we do), then I'm afraid the only viable solution is to indicate somewhere "on the html, right click -> view image" is the best. Low-tech, ok, but it works and is future-proof.


Last.fm to csv freezes after 600 fetched pages

I have tried to download my listening history of 1,164 pages several times now in different browsers on several machines, and it looks like the site has a memory leak, as I only have one browser tab open, and the task manager shows 750 MB RAM usage only for Last.fm to csv after about 600 pages. Sometimes it gets to about 800 pages, but is very slow then until I have to close the browser. Or is there some kind of timeout now maybe, either in the API or in your tool?
My Last.fm username is hjbardenhagen, by the way, and I didn't have any problems fetching my listening history about a year ago which was about 15% smaller then.

@hans-juergen ah, sorry - I've been updating it to work differently (the new version stores responses so they don't have to be downloaded again).
I've rolled back the site to the old version; so hopefully it should work for you again.  I'll make sure I do some proper testing of the new version when I work more on it.
Thanks so much for logging the issue, sorry that it wasn't working for you (totally my fault).

@hans-juergen out of interest - what are you using your data for?  I'm giving a talk next week, and I'd love to have some examples of what people can do when they have access to their own data.

Hi Ben, thanks for your quick reply, I'm trying again now in Firefox 40.0.3 on Windows 10.
Your tool is very useful, thank you very much for creating and releasing it. :-) The official data exporter on the old Last.fm site did not work for everybody, so it was my only chance to have my complete listening history in a nicely formatted CSV file which the online tool Ivy could use to create large Spotify playlists from it and so recreate Last.fm's library radio in the Spotify desktop client. I have written a short description how to do this in a pinned thread of the old Last.fm for Spotify support group:
http://www.last.fm/group/Last.fm+for+Spotify/forum/1249115/_/2235908
Furthermore this method is still or again of interest for other users, because the latest Spotify versions 1.0.12 and 1.0.13 have problems to play a Last.fm radio station continuously, they stop after one track now when you start them on the new Last.fm site with a play link. That is why I have created a new related thread yesterday in the new forum:
https://getsatisfaction.com/lastfm/topics/avoiding-current-spotify-problem-with-playing-only-one-track
OK, fetching my listening history has finished now, 23 errors, but all pages included, as it seems. Thanks again for fixing it so fast.


Fix documentation bug

Fix #196, #165.

Codecov Report

Merging #201 (565600a) into master (3a9f2d1) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master     #201   +/-   ##
=========================================
  Coverage     56.14%   56.14%
  Complexity      728      728
=========================================
  Files           158      158
  Lines          3147     3147
  Branches        345      345
=========================================
  Hits           1767     1767
  Misses         1263     1263
  Partials        117      117

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3a9f2d1...565600a. Read the comment docs.


Release 0.6.0

@robert-bor Can you push a release for 0.6.0?  Not a huge amount of changes but it's jdk 10 compliant and still targets jdk 6 runtime.

Done.


No manifest.text when using Webpack v2.1-beta25

Apologies, I haven't dug into this very deeply, but it seems that when using WP 2.1, using includeManifest: true does not actually add the text…

+1 Faced same issue

@wmertens did you found the workaround of this ?

Instead of doing includeManifest: true you need to pass the name of the manifest chunk:
includeManifest: 'manifest'

It's also worth noting it doesn't currently work when you have sourcemaps but there is PR #56 to fix that.

Instead of doing includeManifest: true you need to pass the name of the manifest chunk:
includeManifest: 'manifest'

It's also worth noting it doesn't currently work when you have sourcemaps but there is PR #56 to fix that.

This was fixed in 3.5.1.


Remove HomePageViewController code

Task/Issue URL: https://app.asana.com/0/72649045549333/1208547223289954/f
Tech Design URL:
CC: @brindy
Description:
This is a follow up after https://github.com/duckduckgo/iOS/pull/3453.
Removes unused code related to HomePageViewController. Home renderers abstraction was also removed, though the implementation itself is used in both Bookmarks controller and to show Favorites while editing URL on non-home tab.
Steps to test this PR:

Make sure code compiles and tests pass.

Definition of Done (Internal Only):

[ ] Does this PR satisfy our Definition of Done?

Copy Testing:

[ ] Use of correct apostrophes in new copy, ie ’ rather than '

Orientation Testing:

[ ] Portrait
[ ] Landscape

Device Testing:

[ ] iPhone SE (1st Gen)
[ ] iPhone 8
[ ] iPhone X
[ ] iPhone 14 Pro
[ ] iPad

OS Testing:

[ ] iOS 15
[ ] iOS 16
[ ] iOS 17

Theme Testing:

[ ] Light theme
[ ] Dark theme

Internal references:
Software Engineering Expectations
Technical Design Template

Hi @dus7 I have no context about these changes and the task is empty, could you provide some info and testing steps?
I see business logic and UI changes, should the PR contain the steps for testing the involved features?

Hi @dus7 I have no context about these changes and the task is empty, could you provide some info and testing steps? I see business logic and UI changes, should the PR contain the steps for testing the involved features?

Hi, I added a few inline comments and an initial PR link in the description. This PR does not cause any UI and/or logic changes. All the modifications were done in previous one (https://github.com/duckduckgo/iOS/pull/3453). This is just removing unused code.

LGTM Just a question: I see that we removed a few translated strings, should we run a translation step in order to remove the strings from the Smartling DB? did you already do it?

Thanks. I tried, but I can't do it since there are no "new" strings and I'm not able to continue in Smartling. It should sort itself out with next translations job I presume.


Potential ban on Blockchain Network due to using unknown accounts in Aggregate Transactions

As discussed with @evias users shouldn't be using unknown accounts in Aggregate transactions right after they create the account. "Unknown to network" means that account has not sent any transactions yet. If that account takes place in an aggregate transaction as a cosigner before it has any confirmed transactions, Blockchain network will ban that account if that happens 5 times repeatedly.
So we must inform the user when he tries to add an unknown account both with address and public key and show him the steps to prevent such a scenario(ask him to create an empty transaction before he proceeds etc.)

As discussed with @Wayonb this concern is not valid after the 0.96 release.

[Wayon Blair] This should not be an issue anymore since 9.6 release.  The problem was for unknown accounts, the public key is unknown.  Public key was required for multisig cosigners.  With 9.6 Multisig account uses the account address which is always available.


New Feature: #1342 Get an internal IP of cloud hosted machine

Functional implementation for GCE, placeholders for the rest at the moment, as discussed.

Squashed the commits, I heard you guys like them

Hi, the PR has been sitting green for over a month, now has a merge conflict. I am happy to resolve the conflict once again, but please let me know if there's a reason the PR wasn't merged (so I can address).

@nathanleclaire ping

It's too much stuff.  Sorry @aperepel, I appreciate the effort, but features of this size and scope need to be discussed in an issue / proposal first before they will get merged in.

Also would like to say sorry for the lack of responsiveness prior to this!

This feature would be really useful. Is there any work going on in order to make this happen @nathanleclaire? :+1:

@schickling No current development efforts but I agree it'd be a really useful feature to have and something we will consider strongly for machine v2.

@nathanleclaire Is there an official roadmap for v2?

@schickling Nothing official so far, it's just been something I've been thinking about.


Correct regex

The current regex fails to work with single digit version numbers. The docker images is trying to download a version of 0.13.0< where the < is coming from the HTML.
This PR updates the regex to not do that anymore!
New regex:

Can you show what was happening? The old regex should only get numbers?

@elduddz Current regex:

This is better, uses the built in digit finding


Move backend processing back from Paphlagon to BEVi

Now that I've done the RAM upgrade, we should restore the data processing workflow on the cluster

We'll need to install R package dependencies to handle the number of infections model, which hasn't been run on BEVi before.

[gopigugan@BEVi ~]$ python3 --version
Python 3.6.8
[gopigugan@BEVi ~]$ R --version
R version 3.6.0 (2019-04-26) -- "Planting of a Tree"
Copyright (C) 2019 The R Foundation for Statistical Computing
Platform: x86_64-redhat-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under the terms of the
GNU General Public License versions 2 or 3.
For more information about these matters see
https://www.gnu.org/licenses/.

We are currently running older versions of Python and R. Should these be updated on BEVi?

My subscription for the clusterware system has expired so I cannot use the package manager to update python on BEVi.  The easiest workaround would probably be to do a local installation (i.e. into /usr/local/bin) of a newer version of Python and make sure that it is in the $PATH.

Updated R to version 4.3 at /usr/local
Python updated to 3.11 at /usr/local
ran into SSL certificate issues downloading packages

@GopiGugan to run some tests on BEVi before we switch back over

Running into issues installing the tidyquant R package on BEVi:
[gopigugan@BEVi ~]$ R -e "install.packages('tidyquant',dependencies=TRUE, repos='http://cran.rstudio.com/')"
...
ERROR: dependency ‘textshaping’ is not available for package ‘ragg’
* removing ‘/home/covizu/.local/lib64/R/library/ragg’
...
ERROR: dependency ‘ragg’ is not available for package ‘tidyverse’
...
ERROR: dependency ‘tidyverse’ is not available for package ‘tidyquant’

Looks like there are some dependencies not available for packages on version 4.3.2 of R. We are currently using version 4.2.2 on Paphlagon.
Downgrading R from version 4.3.2 to 4.2.2 on BEVi

gcc v4.8.5 is the latest supported version on the package manager
R package tidyquant has dependencies that are failing to install because gcc is too old. Compiled gcc version 11.1.0 locally (/usr/local), however gfortran missing. Reinstalling gcc with the following:

./configure --prefix=/usr/local --enable-languages=c,c++,fortran --disable-multilib
make
make install

Issue seems to be the following: R was not built as a library
Reinstalling R version 4.2.2 with the --enable-R-shlib option:
make clean
./configure --prefix=/usr/local --enable-R-shlib
make
make install

rpy2 successfully installed but error importing rpy2
# python3
Python 3.11.3 (main, Jan 16 2024, 01:12:27) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from rpy2.robjects import pandas2ri
Error in glue(.Internal(R.home()), "library", "base", "R", "base", sep = .Platform$file.sep) :
  4 arguments passed to .Internal(paste) which requires 3
Error: could not find function "attach"
Error: object '.ArgsEnv' not found
Fatal error: unable to initialize the JIT

Had to set the following variable to resolve error: export LD_LIBRARY_PATH="$(python3 -m rpy2.situation LD_LIBRARY_PATH)":${LD_LIBRARY_PATH}

Pipeline ran successfully with the test data file:
[covizu@BEVi covizu]$ python3 batch.py --dry-run --infile dev.2000.json.xz
🏄 [0:00:01.038814] Processing GISAID feed data
🏄 [0:00:03.346096] aligned 0 records
🏄 [0:00:03.430148] filtered 1066 problematic features
🏄 [0:00:03.430193]          671 genomes with excess missing sites
🏄 [0:00:03.430204]          163 genomes with excess divergence
🏄 [0:00:03.430838] Parsing Pango lineage designations
🏄 [0:00:05.122239] Identifying lineage representative genomes
🏄 [0:00:05.185900] Reconstructing tree with fasttree2
FastTree Version 2.1.11 Double precision (No SSE3)
...
🏄 [0:01:58.282415][5/56] starting BA.2.1
🏄 [0:02:04.622877][0/56] starting BF.7.5
🏄 [0:02:04.949943][0/56] starting BA.5.1.3
🏄 [0:02:05.022366] Parsing output files
R[write to console]: In addition:
R[write to console]: There were 50 or more warnings (use warnings() to see the first 50)
R[write to console]:

R[write to console]: In addition:
R[write to console]: There were 50 or more warnings (use warnings() to see the first 50)
R[write to console]:

🏄 [0:03:30.864626] All done!

Initiated a dry run to verify there are no issues: nohup python3 batch.py --dry-run > ~/iss499.log &

@GopiGugan reports a successful run

where are we going to store the database?
BEVi has a couple of RAIDs:

/dev/md126 is mounted at /home and is a RAID1 of two drives for 1.8TB storage (790GB currently available)
/dev/md127 is mounted at /data and is a RAID5 of four drives for 11TB storage (9.8 TB currently available)

I don't think there would be a latency difference in writing to one RAID versus another, other than the performance hit of different formats.

current estimate for database size is on the order of 10 GB
I think we can get away with storing it on /home for now for the improved write performance of RAID1 over RAID5
can do database dump backups to /data

Obviously this is on hold until we can get the damn cluster back online (#516)

Currently building database on BEVi (#493, #485)

Investigating a KeyError while building the database:
Traceback (most recent call last):
  File "/home/covizu/covizu/batch.py", line 250, in <module>
    by_lineage = process_feed(args, cur, cb.callback)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/covizu/covizu/batch.py", line 179, in process_feed
    return gisaid_utils.sort_by_lineage(filtered, callback=callback)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/covizu/covizu/covizu/utils/gisaid_utils.py", line 277, in sort_by_lineage
    for i, record in enumerate(records):
  File "/home/covizu/covizu/covizu/utils/gisaid_utils.py", line 220, in filter_problematic
    for record in records:
  File "/home/covizu/covizu/covizu/utils/gisaid_utils.py", line 179, in extract_features
    record = new_records[qname]
             ~~~~~~~~~~~^^^^^^^
KeyError: 'hCoV-19/South'

The issue is that when there is a space in the virus name (qname), e.g.  hCoV-19/South Africa/..... it gets cut off in the minimap2 output:
https://github.com/PoonLab/covizu/blob/db11b2f0d0776064a63a87ff27c11fcf3a01d552/covizu/minimap2.py#L67-L72
So the output was failing when trying to retrieve a record by qname:
https://github.com/PoonLab/covizu/blob/db11b2f0d0776064a63a87ff27c11fcf3a01d552/covizu/utils/gisaid_utils.py#L177-L180
Pipeline is also failing because we are retrieving records and inserting records into the database based on the qname instead of the accession id and qname is not unique:
https://github.com/PoonLab/covizu/blob/db11b2f0d0776064a63a87ff27c11fcf3a01d552/covizu/utils/gisaid_utils.py#L122-L124
https://github.com/PoonLab/covizu/blob/db11b2f0d0776064a63a87ff27c11fcf3a01d552/covizu/utils/gisaid_utils.py#L179-L187

Let's write database dumps to the filesystem on the following basis:

weekly (with every run)
erase 3 weeks out of 4 past three months (retain monthly dumps beyond 3 months to present)
in the long run (?) retain quarterly dumps beyond 3 years

@GopiGugan testing out script for clearing out expired logs

@GopiGugan to push the clean up script to repo and close


tagHandling: 'html' and ignoreTags don't seem to work together

Describe the bug
We have a list of words we don't want translated. We tag the words before sending them to Deepl with the tag we configure in the ignoreTags property. This works well when tagHandling is set to XML but not when it's set to HML

Ah it's a clear case of RTFM :-)
Thanks for the explanation.


Change readme instructions to sudo snap refresh --edge qemu-virgil

Change readme instructions to sudo snap refresh --edge qemu-virgil
The removable-media  plug is not in stable snap yet.
https://github.com/ogra1/qemu-virgil-snap/issues/5#issuecomment-601703741

well, it will soon be in stable i wont change the instructions for this ... since it is a temporary condition :)

Well it's better since you have the latest version (4.2.0) and so you can use the documentation to add more parameters when launching QEMU.


How to test oracle with YCSB

How to test oracle with YCSB
1 Which jar can be used for testing oracle
2 How to config the oralce workload
3 How to config oracle properties in workload, and use which db.driver

Oracle makes a lot of products. Which one do you mean?

I want to test oracle/database:<IP_ADDRESS>-se2   with YCSB

for their database offering, you should use the JDBC client binding along with their JDBC driver.

It's OK now , I use ojdbc8.jar, and my config like this:


  I think the destructive action warrants its own distinctive button, maybe a nice bright red "danger" button.
Or at a minimum change the dropdown to a button group with a separator and the delete button use the danger css

This is not permanent. There is not a set place for the delete button yet, but it probably won't be where it is now.

What if I accidently delete a linode, is there any way to get back up?

The same deal as it is now, you'll be warned before you actually commit to it and if you do it anyway, you have a small window during which you can reach out to support and we can fix it for you.

What is that window like? how much time?

It varies, I'm not sure. That decision doesn't lie with the manager team, though.

Okay, thanks for the help. I really appreciate it. I hope I can get back the linode I lost.

If you lost a Linode in production, reach out to support immediately. If you lost a Linode in the alpha, you'll just have to live with it being gone.

We'll be going forward with this by adding a delete action to the Linode action dropdown. See https://github.com/linode/manager/issues/582

Thanks for the support :)


DirectRabbitListenerContainerFactory not instrumented for tracing, whereas SimpleRabbitListenerContainerFactory is

Hi,
I've noticed that some of our services aren't logging Sleuth trace ids. When making a comparison I found that those just annotating listener methods with
@RabbitListener(queues = "...")
are working as expected, whereas those using
@RabbitListener(queues = "...", containerFactory = "someContainerFactory")
where someContainerFactory returns an instance of DirectRabbitListenerContainerFactory
are not adding trace ids when logging from invocations of those listener methods.
Investigating further, I saw that org.springframework.cloud.sleuth.instrument.messagingTraceMessagingAutoConfiguration  is decorating org.springframework.amqp.rabbit.config.SimpleRabbitListenerContainerFactory (which I believe is the default), but not the newer DirectRabbitListenerContainerFactory in the same package.
Is it deliberately not decorated, or has the sleuth project just not caught up with the rabbit support yet?

Is it deliberately not decorated, or has the sleuth project just not caught up with the rabbit support yet?

I guess we haven't caught up. Are you interested in filing a PR to fix this? You can file it against the 2.2.x branch and I'll then apply it to master.

Actually this first needs to be fixed in Brave. Can you please file an issue there? The brave.spring.rabbit.SpringRabbitTracing class can only instrument SimpleRabbitListenerContainerFactory.

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

Closing due to lack of requested feedback. If you would like us to look at this issue, please provide the requested information and we will re-open the issue.


Add a fitler parsing function returning Result instead of writing errors to stderr

I suggest adding a new method env_filter::filter::try_parse(&mut self, filters: &str) -> Result<&mut Self, ParsingError>. This would imply adding a new ParsingError type and a small refactoring, but would make the library more convenient in some situations.
In my use case, I don't have access to the contents of stderr when the application is starting, and I actually don't want the application to start if the logger is configured incorrectly. Such a method would let me do this without copying the parser implementation to the application code.
If the maintainers agree that this can be added to the env_logger, I am happy to create a PR.

I think its a reasonable addition.


Use gmm workloads with different values of d.

I had to modify the data generator to get this to work.

Debugging the tensorflow failure will be tricky, as the log file is so large that GitHub will not even let me download it. I suspect it may be time to not print all the inputs and results in the log.
My hunch is that the tensorflow failure is due to OOM, but I ran it locally and it doesn't look excessive to me.


False positive alert in dump ledger state job

The dump ledger state job begins by requesting the latest snapshot ledger.
https://github.com/stellar/go/blob/master/exp/tools/dump-ledger-state/run_test.sh#L12
When this request fails LATEST_LEDGER is set to the empty string which causes the job to fail and trigger an alert.
We should abort the job without triggering an alert if LATEST_LEDGER is empty.
Also, we should avoid sending slack alerts from the script. Instead we should log an error message
in the script on failures. Then we can create an elastic search rule to trigger the alert.

Moving to current sprint. Looks like easy to fix and will remove false positives.

Fixed in https://github.com/stellar/go/pull/2822.


Image extraction problem

Hello,
I get error for each image extracted :
Starting the application... Initialized model and FAISS index with dimension 768 Starting Streamlit UI Loading FAISS index and metadata Application finished 2024-10-24 23:28:25.259 Examining the path of torch.classes raised: Tried to instantiate class '__path__._path', but it does not exist! Ensure that it is registered via torch::class_ Starting the application... Initialized model and FAISS index with dimension 768 Starting Streamlit UI Indexing documents in /Users/ilan/_INFOSTRATES/_AI/LaboP9/data Indexing documents in directory: /Users/ilan/_INFOSTRATES/_AI/LaboP9/data Processing file: /Users/ilan/_INFOSTRATES/_AI/LaboP9/data/Duval-PantagruelsGenealogyRedemptive-1984fr.pdf Reading PDF: /Users/ilan/_INFOSTRATES/_AI/LaboP9/data/Duval-PantagruelsGenealogyRedemptive-1984fr.pdf Chunking text of length 102469 with chunk size 500 and overlap 50 Created 35 chunks Processing image Processing image Processing image Processing image Processing file: /Users/ilan/_INFOSTRATES/_AI/LaboP9/data/GoffartVisionsubmitted.pdf Reading PDF: /Users/ilan/_INFOSTRATES/_AI/LaboP9/data/GoffartVisionsubmitted.pdf Chunking text of length 161144 with chunk size 500 and overlap 50 Created 55 chunks Processing image Processing image Processing image Processing image Processing image Processing image Processing image Processing image Processing image Processing image Processing image Processing file: /Users/ilan/_INFOSTRATES/_AI/LaboP9/data/Astrobioethicsfr.pdf Reading PDF: /Users/ilan/_INFOSTRATES/_AI/LaboP9/data/Astrobioethicsfr.pdf Chunking text of length 40677 with chunk size 500 and overlap 50 Created 13 chunks Processing image Processing file: /Users/ilan/_INFOSTRATES/_AI/LaboP9/data/articlescientifiqueIlan.pdf Reading PDF: /Users/ilan/_INFOSTRATES/_AI/LaboP9/data/articlescientifiqueIlan.pdf Chunking text of length 94517 with chunk size 500 and overlap 50 Created 34 chunks Processing image Processing image Processing file: /Users/ilan/_INFOSTRATES/_AI/LaboP9/data/Tombinsetal2024.pdf Reading PDF: /Users/ilan/_INFOSTRATES/_AI/LaboP9/data/Tombinsetal2024.pdf Chunking text of length 63371 with chunk size 500 and overlap 50 Created 22 chunks Processing image Processing image Processing image Processing image Processing image Processing image Processing image Processing image Processing image Encoding 186 document chunks and images 2024-10-24 23:28:35.518 Uncaught app exception Traceback (most recent call last):   File "/opt/miniconda3/envs/labop9/lib/python3.11/site-packages/streamlit/runtime/scriptrunner/exec_code.py", line 88, in exec_func_with_error_handling     result = func()              ^^^^^^   File "/opt/miniconda3/envs/labop9/lib/python3.11/site-packages/streamlit/runtime/scriptrunner/script_runner.py", line 579, in code_to_exec     exec(code, module.__dict__)   File "/Users/ilan/_INFOSTRATES/_AI/LaboP9/local_genai_search.py", line 477, in <module>     main()   File "/Users/ilan/_INFOSTRATES/_AI/LaboP9/local_genai_search.py", line 388, in main     index_documents(documents_path)   File "/Users/ilan/_INFOSTRATES/_AI/LaboP9/local_genai_search.py", line 164, in index_documents     image_description = llava_generate("Describe this image in detail.", doc)                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^   File "/Users/ilan/_INFOSTRATES/_AI/LaboP9/local_genai_search.py", line 294, in llava_generate     base64_image = encode_image_to_base64(image_path)                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^   File "/Users/ilan/_INFOSTRATES/_AI/LaboP9/local_genai_search.py", line 289, in encode_image_to_base64     with open(image_path, "rb") as image_file:          ^^^^^^^^^^^^^^^^^^^^^^ ValueError: embedded null byte 2024-10-24 23:28:35.602 Examining the path of torch.classes raised: Tried to instantiate class '__path__._path', but it does not exist! Ensure that it is registered via torch::class_
Any clue ?
Thanks you

I tried a simple test to verify my pdf and all works fine
`import os
import fitz  # PyMuPDF for PDF image extraction
from PIL import Image
import io
import base64
def extract_images_from_pdf(file_path):
images = []
# Open the PDF file
doc = fitz.open(file_path)
try:
for page_num in range(len(doc)):
page = doc.load_page(page_num)
# Retrieve list of images stored in the page
image_list = page.get_images(full=True)
for img_index, img in enumerate(image_list):
xref = img[0]
# Extract the image metadata using xref index
base_image = doc.extract_image(xref)
# Get the image bytes
image_bytes = base_image["image"]
# Create an image object and add to the list
image = Image.open(io.BytesIO(image_bytes))
images.append((file_path, page_num + 1, img_index, image))
finally:
# Ensure the document is closed to free resources
doc.close()
return images
def encode_image_to_base64(image):
buffered = io.BytesIO()
image.save(buffered, format="PNG")
img_str = base64.b64encode(buffered.getvalue()).decode('utf-8')
return img_str
Example usage
pdf_path = "/Users/ilan/_INFOSTRATES/_AI/LaboP9/data/Tombinsetal2024.pdf"
images = extract_images_from_pdf(pdf_path)
if images:
for file_path, page_num, img_index, image in images:
print(f"Found image in {file_path}, Page: {page_num}, Index: {img_index}")
    # Encode the image to base64
    base64_image = encode_image_to_base64(image)
    print(f"Base64 encoded image for page {page_num} and index {img_index}: {base64_image[:100]}... (truncated)")

else:
print("No images found in the PDF.")`
(labop9) ilan@macbook-pro LaboP9 % python app.py

Found image in /Users/ilan/_INFOSTRATES/_AI/LaboP9/data/Tombinsetal2024.pdf, Page: 1, Index: 0
Found image in /Users/ilan/_INFOSTRATES/_AI/LaboP9/data/Tombinsetal2024.pdf, Page: 1, Index: 1
Found image in /Users/ilan/_INFOSTRATES/_AI/LaboP9/data/Tombinsetal2024.pdf, Page: 2, Index: 0
Found image in /Users/ilan/_INFOSTRATES/_AI/LaboP9/data/Tombinsetal2024.pdf, Page: 3, Index: 0
Found image in /Users/ilan/_INFOSTRATES/_AI/LaboP9/data/Tombinsetal2024.pdf, Page: 4, Index: 0
Found image in /Users/ilan/_INFOSTRATES/_AI/LaboP9/data/Tombinsetal2024.pdf, Page: 5, Index: 0
Found image in /Users/ilan/_INFOSTRATES/_AI/LaboP9/data/Tombinsetal2024.pdf, Page: 6, Index: 0
Found image in /Users/ilan/_INFOSTRATES/_AI/LaboP9/data/Tombinsetal2024.pdf, Page: 6, Index: 1
Found image in /Users/ilan/_INFOSTRATES/_AI/LaboP9/data/Tombinsetal2024.pdf, Page: 7, Index: 0
(labop9) ilan@macbook-pro LaboP9 % python app.py
Found image in /Users/ilan/_INFOSTRATES/_AI/LaboP9/data/Tombinsetal2024.pdf, Page: 1, Index: 0
Base64 encoded image for page 1 and index 0: iVBORw0KGgoAAAANSUhEUgAAAPgAAAEPCAIAAADgU55JAAEAAElEQVR4nJz9d1yTadc1gN4pkEYCSQihJpTQe++C9KZS7IrYu87o... (truncated)
Found image in /Users/ilan/_INFOSTRATES/_AI/LaboP9/data/Tombinsetal2024.pdf, Page: 1, Index: 1
Base64 encoded image for page 1 and index 1: iVBORw0KGgoAAAANSUhEUgAAAOwAAAEqCAIAAACZdyteAAEAAElEQVR4nOy9d5xkV3Un/j03vPcqdHWcnhwVRjmgjBBIQuRkwLIx... (truncated)
Found image in /Users/ilan/_INFOSTRATES/_AI/LaboP9/data/Tombinsetal2024.pdf, Page: 2, Index: 0
Base64 encoded image for page 2 and index 0: iVBORw0KGgoAAAANSUhEUgAABkAAAAPQCAIAAADhMCAUAAEAAElEQVR4nOz9d7wt11UljI4511oV9t4n3Xyvrq5yzpZly3JExhEB... (truncated)
Found image in /Users/ilan/_INFOSTRATES/_AI/LaboP9/data/Tombinsetal2024.pdf, Page: 3, Index: 0
Base64 encoded image for page 3 and index 0: iVBORw0KGgoAAAANSUhEUgAAB2wAAATMCAIAAADEbODkAAEAAElEQVR4nOx9d7wU1fn385yZ2XJ3b+8XuHSQJh1RQBQsKPZg76lG... (truncated)
Found image in /Users/ilan/_INFOSTRATES/_AI/LaboP9/data/Tombinsetal2024.pdf, Page: 4, Index: 0
Base64 encoded image for page 4 and index 0: iVBORw0KGgoAAAANSUhEUgAABtYAAAd5CAIAAADn2byPAAEAAElEQVR4nOzdd5wb13Uo/nOnomMBbO+Vy7bsvYukCkk1Ky6SbNmS... (truncated)
Found image in /Users/ilan/_INFOSTRATES/_AI/LaboP9/data/Tombinsetal2024.pdf, Page: 5, Index: 0
Base64 encoded image for page 5 and index 0: iVBORw0KGgoAAAANSUhEUgAAB6EAAAmOCAIAAABqaq9SAAEAAElEQVR4nOzdd5gc1Zk2/OeESh0nz2iSpNFIM8o5SygihAiSyCAw... (truncated)
Found image in /Users/ilan/_INFOSTRATES/_AI/LaboP9/data/Tombinsetal2024.pdf, Page: 6, Index: 0
Base64 encoded image for page 6 and index 0: iVBORw0KGgoAAAANSUhEUgAABqQAAAP4CAIAAAA0xgpEAAEAAElEQVR4nOzdd3hc130n/HPO7dMrBr33Xth7l0RRvVdbrkmcxClv... (truncated)
Found image in /Users/ilan/_INFOSTRATES/_AI/LaboP9/data/Tombinsetal2024.pdf, Page: 6, Index: 1
Base64 encoded image for page 6 and index 1: iVBORw0KGgoAAAANSUhEUgAACBsAAALmCAIAAACG5pmpAAEAAElEQVR4nOz9d5icx3knilb4Yufp7ulJmBnkQSIJgCTABCZRFKlE... (truncated)
Found image in /Users/ilan/_INFOSTRATES/_AI/LaboP9/data/Tombinsetal2024.pdf, Page: 7, Index: 0
Base64 encoded image for page 7 and index 0: iVBORw0KGgoAAAANSUhEUgAACBsAAASyCAIAAAByKeZyAAEAAElEQVR4nOz9Z5RkR5YeCN57zewJd4+IVAASqExorYEqlIKsQomu... (truncated)

there's an embedded null byte in the image_path, which is causing the open function to fail.
have added fix for this, hope it will work for you


Add travis based on industrial_ci

Surprisingly on this setup all tests succeed while they all fail on our internal CI or on my local pc ...
This could be extended to do more advanced stuff like clang-format, clang-tidy etc

Any particular reason to use moveit_ci?

no, it was simple to copy paste, that's the whole reason

@gavanderhoorn would you prefer industrial_ci or something independent and minimal?

I'm not the maintainer here, but seeing as we use industrial_ci in all other repositories that have CI setup, I think I'd like to keep things consistent and use industrial_ci here as well.

[..] and minimal?

Setup for industrial_ci is pretty concise actually I believe:
sudo: required
dist: trusty
language: generic

cache:
  directories:
    - $HOME/.ccache

git:
  quiet: true

env:
  global:
    - CCACHE_DIR=$HOME/.ccache
    - ROS_REPO=ros
  matrix:
    - ROS_DISTRO=kinetic
    - ROS_DISTRO=kinetic ROS_REPO=ros-shadow-fixed

install:
  - git clone --quiet --depth=1 https://github.com/ros-industrial/industrial_ci.git .ci_config

script:
  - .ci_config/travis.sh

sure, I'll test that later on
by "minimal" I was refering more to having an external dependency (*_ci) vs calling cmake directly as this code seems to be rather standalone. I agree on aiming for consistency thou

@gavanderhoorn @Jmeyer1292  industrial_ci successfully builds and runs the tests, please squash on merge

Seeing as this is @Jmeyer1292's repository and Travis is not enabled for this repository I'll leave review and merging to @Jmeyer1292.

I added a small fix that makes the tests succeed on my system as well

@simonschmeisser wrote:

please squash on merge

@Levi-Armstrong: I believe you missed this.

Could consider a force-push after a rebase to squash things.

It's fine this way, having the auto vs Vector and the missing eigen dependency in,  separate commits also make sense. Thanks for merging!

@gavanderhoorn Sorry missed your comment about squashing.


psycopg2. Programming error -- datcollate

I am running into errors with the latest version of pgcli.
When I do:
> pgcli -hsome_host.net some_database -Uavazquez -p 54321

I get:
Traceback (most recent call last):
  File "/Users/avazquez/anaconda3/envs/py34/bin/pgcli", line 9, in <module>
    load_entry_point('pgcli==0.16.0', 'console_scripts', 'pgcli')()
  File "/Users/avazquez/anaconda3/envs/py34/lib/python3.4/site-packages/click/core.py", line 610, in __call__
    return self.main(*args, **kwargs)
  File "/Users/avazquez/anaconda3/envs/py34/lib/python3.4/site-packages/click/core.py", line 590, in main
    rv = self.invoke(ctx)
  File "/Users/avazquez/anaconda3/envs/py34/lib/python3.4/site-packages/click/core.py", line 782, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/avazquez/anaconda3/envs/py34/lib/python3.4/site-packages/click/core.py", line 416, in invoke
    return callback(*args, **kwargs)
  File "/Users/avazquez/anaconda3/envs/py34/lib/python3.4/site-packages/pgcli/main.py", line 344, in cli
    pgcli.run_cli()
  File "/Users/avazquez/anaconda3/envs/py34/lib/python3.4/site-packages/pgcli/main.py", line 170, in run_cli
    self.refresh_completions()
  File "/Users/avazquez/anaconda3/envs/py34/lib/python3.4/site-packages/pgcli/main.py", line 297, in refresh_completions
    completer.extend_database_names(pgexecute.databases())
  File "/Users/avazquez/anaconda3/envs/py34/lib/python3.4/site-packages/pgcli/pgexecute.py", line 235, in databases
    cur.execute(self.databases_query)
psycopg2.ProgrammingError: column d.datcollate does not exist
LINE 4:        d.datcollate as "Collate",
               ^

This is with psycopg2 v2.6  and pgcli v0.16.
> import psycopg2 as pg
> pg.__version__
'2.6 (dt dec pq3 ext lo64)'

> import pgcli
> pgcli.__version__
'0.16.0'

I am in a python 3.4 conda environment with the latest version of Python:
$ python --version
Python 3.4.2 :: Anaconda 2.1.0 (x86_64)

I installed psycopg2 with pip install pgcli and did not get any errors.
Any thoughts on what could be wrong?

What version of postgres database are you using? pgcli only has support for version 9.2 and above, but there is a pending issue to add support for the older versions.
Did this issue crop up after you upgraded to the new version or is this the first time you're trying pgcli?
I'm trying to find out if this is because of the new additions in the latest version.

Thank you @amjith First time using pgcli. Sorry, I should have mentioned that.  I am working against an older version of a PostgreSQL database, so that probably explains it:
> select version();
PostgreSQL 8.2.15 (Greenplum Database <IP_ADDRESS> build 2) on x86_64-unknown-linux-gnu, compiled by GCC gcc (GCC) 4.4.2 compiled on Oct  7 2013 15:24:14

What issue is the one you mention about adding support to older versions? (so that I can track it). Is it this one? https://github.com/amjith/pgcli/issues/18
Thanks.

Yes that's the one.
The problem is, in order to populate the autocompletion I have to run sql queries against the database to get the list of tables and columns. That sql query is different between the various versions of the Postgres and I decided to stick with 9.2 and above since that's the one I had on my machine. :(
Now I have a few docker instances with older versions, so I'll try and get at least the basic stuff working.
Thanks for reporting the bug. Cheers!

Support for postgres 8.x is now available in v1.0. Please upgrade.
pip install -U pgcli


Integrate pipelining logic

Integrate recently developed pipelining logic that better handles exceptions.

Added exception handling to batch.execute() and in future.whenComplete lambdas.


[FEATURE]  toggle harvester node driver with the harvester global flag

link to https://github.com/rancher/rancher/issues/34886
We should add the Harvester node driver as a built-in type and also support downloading the Harvester node driver within the air-gap env.

Can we help to validate this issue on https://github.com/rancher/rancher/issues/34886 thanks? and both can be closed after passing the validation.

Verified fixed on Rancher v2.6.3-rc1 with builtin node driver
Pass the following test scenario. Close this issue and rancher/rancher#34886 accordingly.
Test in Rancher v2.6.3-rc1 under airgapped environment

[x] Harvester feature flag will be enabled by default and turned on harvester node driver accordingly

[x] If the feature flag was turned off, nothing will change to the Harvester node driver.

[x] Enable/disable the Harvester node driver will not affect the state of the feature flag.

[x] If the feature flag was turned on, we enable the Harvester node driver automatically

Result

Harvester feature flag are turned on by default

The node driver also enabled automatically

Deactivate the  feature flag,

Harvester node driver will not be affected

Deactivate harvester node driver

Harvester feature flag did not being affected

Activate harvester node driver

Harvester feature flag did not being affected

Both harvester feature flag and node driver are deactivated, Activate feature flag will also bring up harvester node driver

Test Information

Rancher version: v2.6.3-rc1

Environment Preparation

Prepare an airgapped ubuntu vm
Install rancher 2.6.3-rc1

Verify Steps

Environment preparation as above steps
Open global setting -> feature flag in rancher
Check harvester feature flag
Open cluster management -> Driver page
Check harvester node driver
Deactivate harvester feature flag
Activate harvester feature flag
Deactivate harvester node driver
Activate harvester node driver
Deactivate both harvester flag and node driver
Activate harvester feature flag


Fix spelling mistake

@BartoszMiller Please sign the Contributor License Agreement!
Click here to manually synchronize the status of this Pull Request.
See the FAQ for frequently asked questions.

@BartoszMiller Thank you for signing the Contributor License Agreement!


Node: Permission denied on filesystem via Bitbar, Works in Terminal

I have a Node.js plugin, that reads and writes from/to the filesystem, using something like this:
fs.writeFileSync('example.txt', 'Test123');
When I make the plugin executable and run it on the terminal, everything works fine. However when I want to run it as a Bitbar plugin (no changes, exactly the same file), i get
ERROR: ACCES: permission denied, open 'example.txt'
Yes, the directory in which the plugin is located has write privileges for all users. Any other ideas, where this might come from?

Can you post your script?

Sure thing @oleander. This is my test script:
`#!/usr/bin/env /usr/local/bin/node
var fs = require('fs');
const FILENAME = 'file.txt';
function writeToFile(high, low)
{
fs.writeFileSync(FILENAME, "TEST");
console.log('written');
}
writeExtremaToFile(0,0);
`
Running this in the command line via node script.1m.js works fine. However, running this as a Bitbar plugin will result in a ERROR: EACCES: permission denied, open 'file.txt'. I've tried all chmod settings, and even with 777 applied to both the directory and the file, I still get the same error.

Try replacing FILENAME with the absolute path to the file and see if you get the same error message.

@oleander Thank you so much! It's working now. I guess I could have tried that myself, seems to make sense.


@logan-markewich Mixture of agents issue

          @logan-markewich Thanks for Responding.

I have Azure openai  models (3.5 and 4o), how can i use these models inside the Custom model as Mixture of agents?
Originally posted by @Neethan54 in https://github.com/run-llama/llama_index/issues/14962#issuecomment-2252303398

@Neethan54 you can follow the guide I linked
https://docs.llamaindex.ai/en/stable/module_guides/models/llms/usage_custom/#example-using-a-custom-llm-model-advanced
You can implement mixture of agents inside a custom llm. That custom llm class can use multiple openai instances. You can even just run the mixture of agents pack inside of the custom llm if you don't want to reimplement the logic
from typing import Optional, List, Mapping, Any

from llama_index.core import SimpleDirectoryReader, SummaryIndex
from llama_index.core.callbacks import CallbackManager
from llama_index.core.llms import (
    CustomLLM,
    CompletionResponse,
    CompletionResponseGen,
    LLMMetadata,
)
from llama_index.core.llms.callbacks import llm_completion_callback
from llama_index.core import Settings

class OurLLM(CustomLLM):
    context_window: int = 32000
    num_output: int = 512
    model_name: str = "custom"
    mixture_of_agents = MixtureOfAgentsPack(...)

    @property
    def metadata(self) -> LLMMetadata:
        """Get LLM metadata."""
        return LLMMetadata(
            context_window=self.context_window,
            num_output=self.num_output,
            model_name=self.model_name,
        )

    @llm_completion_callback()
    def complete(self, prompt: str, **kwargs: Any) -> CompletionResponse:
        response = self.mixture_of_agents.run(prompt)
        return CompletionResponse(text=response)

    @llm_completion_callback()
    def stream_complete(
        self, prompt: str, **kwargs: Any
    ) -> CompletionResponseGen:
        raise NotImplementedError

Hi @logan-markewich , Thanks  for helping , But constantly im getting "Event loop is closed" error from the react agent, when i was customllm as llm.
How can i solve this?
Thanks in advance


Define quality gates and link to versioning schema

Currently all resource providers are versioned at 0.1.0.
Would be good to have a set of quality gates that can be implemented and reflect in the versioning scheme
Quality gates could be things like: support all methods, unit testing, documentation, review, contract testing.
Quality gates can also be also explicitly discussed, documented and deviated from deviated from.

I realized that we will need probably to use GithubFlow (similar to what is suggested here: https://hackernoon.com/a-monorepo-github-flow-and-automation-ftw-c41a2d9c48bb). Github Releases as our source of truth that will be sync automatically to S3 bucket. All of that can be automated in Actions while being triggered by a PR.

From the perspective of coverage, we a have minimum set in jest config file and 5% variation allowed by CodeCov (in its configuration).

duplicate of #31


Prepare project for packaging

Add a setup.py
Add a service file for systemd
Add production docker file
Add venv installation script that installs integreat-cms and
all dependencies.
Add *.egg-info to gitignore
Run setup.py in Dockerfile
docker-compose only needs to start the CMS

This continues the discussion in PR #108

Building .deb packages is documented at https://en.opensuse.org/openSUSE:Build_Service_Debian_builds

Packaging works in a virtual environment:
python3 -m venv .venv
source .venv/bin/activate
pip3 install stdeb
python3 setup.py --command-packages=stdeb.command bdist_deb