Question: What's the best way to read large JSON files directly into LiteDB

Here's my scenario, I need to insert around 6 million records from a flat json file into LiteDB.  First off, fantastic work on this project, very impressive.  Next, I can read directly from disc, convert into a BsonDocument and insert the records at 1000 entries per BsonDocument array.  This get's exponentially slower with each consecutive insert.  Is there a faster more optimized way to do this?  I'm able to do the same thing in MongoDB using mongoimport and the throughput appears to be substantially higher.  Any assistance here would be greatly appreciated, thanks.

Hi @godlikemouse, thanks! Import/Export features will be avaiable in next 3.2 version.
For now, try:

Read JSON file as an array (file must be a json array). JsonSerializer support read array on deman (returning an IEnumerable<BsonValue>): https://github.com/mbdavid/LiteDB/blob/master/LiteDB/Document/Json/JsonSerializer.cs#L108
Buffer documents in batch (like 5000 documents) and then run 1 insert command without transaction (transaction will me auto-created to this 5000 documents). If is a single operation, you can also disable journal to be faster.

Hi @mbdavid,
Thank you for the prompt response.
I've already disabled journal.
LiteDatabase db = new LiteDatabase("Filename=test.db; Journal=false"
I'll work on trying to read the values from the JSON file into an IEnumerable instead of an IEnumerable and see if that speeds things up.  I can change the batch to 5000 easy enough.

Inside LiteDB there is an Batch IEnumerable method extensions you can use (actually, copy because is internal): https://github.com/mbdavid/LiteDB/blob/master/LiteDB/Utils/Extensions/LinqExtensions.cs
using(var engine = new LiteEngine("...;journal=false"))

var reader = new TextReader(...);

foreach(var docs in JsonSerializer.DeserializeArray(reader).Batch(5000));
{
    engine.Insert("mycol", docs)
}

Oh, very nice, let me give that a try.  Thank you.

Suggest adding this into Wiki/documents

Hi @mbdavid,
I've been working to speed up the inserts based on the information you've given me using a smaller control set.  One thing I've noticed is that when I bulk insert the entries in groups of 5000, everything inserts well until around the 25000 entry range.  From that point forward the performance degradation is exponential.  Doing further experiments, I found that if I create a new collection every 25000 entries, performance remains relatively constant getting all 64495 entries in rather quickly.  However, when batching at intervals of 5000 to the same collection, these are the numbers I'm seeing:
Inserted 5000 records : 5000 total, elapsed time: 00:00:00.8136724, total elapsed time 00:00:02.1729355.
Inserted 5000 records : 10000 total, elapsed time: 00:00:01.1526203, total elapsed time 00:00:04.2712472.
Inserted 5000 records : 15000 total, elapsed time: 00:00:01.0188746, total elapsed time 00:00:06.5339195.
Inserted 5000 records : 20000 total, elapsed time: 00:00:01.3106744, total elapsed time 00:00:08.8663560.
Inserted 5000 records : 25000 total, elapsed time: 00:00:01.2519512, total elapsed time 00:00:11.1984341.
Inserted 5000 records : 30000 total, elapsed time: 00:00:01.5048521, total elapsed time 00:00:13.4588529.
Inserted 5000 records : 35000 total, elapsed time: 00:00:01.9553882, total elapsed time 00:00:16.6838628.
Inserted 5000 records : 40000 total, elapsed time: 00:00:04.7825360, total elapsed time 00:00:22.4643627.
Inserted 5000 records : 45000 total, elapsed time: 00:00:57.4881217, total elapsed time 00:01:21.0844113.
Inserted 5000 records : 50000 total, elapsed time: 00:00:35.9499825, total elapsed time 00:01:58.0451584.
Inserted 5000 records : 55000 total, elapsed time: 00:01:32.6292249, total elapsed time 00:03:31.9206959.
Inserted 5000 records : 60000 total, elapsed time: 00:01:21.0505327, total elapsed time 00:04:53.9208617.
Inserted 5000 records : 64495 total, elapsed time: 00:01:53.8559244, total elapsed time 00:06:48.5311372.
64495 total records inserted, 00:06:50.5602728 total time.
Also for reference, here is the code I'm executing
            Stopwatch csw = new Stopwatch();
            Stopwatch sw = new Stopwatch();

            csw.Start();

            using (JsonStream stream = new JsonStream(File.OpenRead(path)))
            {
                using (LiteDatabase db = new LiteDatabase("filename=test.db; journal=false"))
                {
                    string importName = string.Format("{0}-import", collectionName);
                    db.DropCollection(importName);
                    LiteCollection<BsonDocument> collection = db.GetCollection(importName);

                    //collection.EnsureIndex("id", true);
                    //collection.EnsureIndex("name");

                    //List<BsonDocument> docs = new List<BsonDocument>();
                    List<BsonValue> docs = new List<BsonValue>();
                    string json = null;
                    while ((json = stream.ReadLine()) != null)
                    {

                        try
                        {

                            //docs.Add(JsonSerializer.Deserialize(json).AsDocument);
                            docs.Add(JsonSerializer.Deserialize(json));

                            if (++count > 0 && count % BULK_LIMIT == 0)
                            {

                                sw.Start();
                                collection.Insert(docs.Select(x => x.AsDocument));
                                sw.Stop();
                                Debug.WriteLine("Inserted {0} records : {1} total, elapsed time: {2}, total elapsed time {3}.", BULK_LIMIT, count, sw.Elapsed, csw.Elapsed);
                                sw.Reset();
                                docs.Clear();
                            }
                        }
                        catch(Exception ex)
                        {
                            Console.Error.WriteLine(ex.Message);
                        }
                    }

                    if (docs.Count > 0)
                    {
                        sw.Start();
                        collection.Insert(docs.Select(x => x.AsDocument));
                        sw.Stop();
                        Debug.WriteLine("Inserted {0} records : {1} total, elapsed time: {2}, total elapsed time {3}.", BULK_LIMIT, count, sw.Elapsed, csw.Elapsed);
                    }

                    /*
                    using(var sr = new StreamReader(filename))
                    {
                        var docs = JsonSerializer.DeserializeArray(sr);

                        collection.Insert(docs.Select(x => x.AsDocument));
                    }
                    */

                    db.DropCollection(collectionName);
                    db.RenameCollection(importName, collectionName);
                }
            }

I tried to follow as closely as I could with the examples you've provided in combination with the source code of the LiteShell bulk import.  Could you please tell me if there's an issue with the way I'm doing things or if there's an optimization approach you could suggest?  The only real difference I see here is that instead of using the Batch() method, I'm queuing up the BsonValue elements in my own list.  Thanks in advance for your help.

As a follow up, when I execute the following:
                    List<BsonDocument> docs = new List<BsonDocument>();
                    string json = null;
                    sw.Start();

                    while((json = stream.ReadLine()) != null)
                    {
                        docs.Add(JsonSerializer.Deserialize(json).AsDocument);
                        count++;
                    }
                    sw.Stop();
                    Debug.WriteLine("Inserted {0} records into list, elapsed time: {1}, total elapsed time {2}.", count, sw.Elapsed, csw.Elapsed);

                    count = docs.Count();
                    collection.Insert(docs.Select(x => x.AsDocument));

                    csw.Stop();

The result is the following:
Inserted 64495 records into list, elapsed time: 00:00:10.8103604, total elapsed time 00:00:10.8637538.
64495 total records inserted, 00:00:34.6382347 total time.
Next, when I implement this:
                    using(var sr = new StreamReader(filename))
                    {
                        sw.Start();
                        var docs = JsonSerializer.DeserializeArray(sr);
                        count = docs.Count();
                        sw.Stop();
                        Debug.WriteLine("Inserted {0} records into list, elapsed time: {1}, total elapsed time {2}.", count, sw.Elapsed, csw.Elapsed);

                        collection.Insert(docs.Select(x => x.AsDocument));
                    }

The result is the following:
Inserted 64495 records into list, elapsed time: 00:00:05.0990274, total elapsed time 00:00:06.6898283.
64495 total records inserted, 00:00:06.7126147 total time.
It would appear that there's an optimization here that I am missing.  Any ideas what I can do to speed this up?  Also it's worth mentioning, that I can't actually use the JsonSerializer.DeserializeArray() in my final project because the actual JSON I will be parsing contains long values which need to be parsed and replaced with {"$numberLong": ""} values.  Thanks.

If you are insert "only" 64k documents you can insert as a single insert an not break into chunks. Thats problably why when you run in a single insert runs in 6s.
You code seams ok, you loose some time to double-deserialize but this is not too long to be huge difference after some inserts. Can you show 1 document to know document size/schema to reproduce. Until this, try close/open datafile in each batch insert (could be page cache memory consume).

Certainly I'll post one of the documents as an example.  In the meantime, I pre-parsed the largest of the files to replace the long values with the $numberLong equivalents, then ran the code you originally shared:
using(TextReader reader = File.OpenText(path))
                    {

                        foreach (var d in JsonSerializer.DeserializeArray(reader).Batch(BULK_LIMIT))
                        {
                            sw.Start();
                            collection.Insert(d.Select(x => x.AsDocument));
                            count += d.Count();
                            sw.Stop();
                            Debug.WriteLine("Inserted {0} records into list, elapsed time: {1}, total elapsed time {2}.", count, sw.Elapsed, csw.Elapsed);
                        }

                    }

This approach however had the same issue, after about 35000 - 40000 records, the time began to grow exponentially:
Inserted 5000 records into list, elapsed time: 00:00:00.8403290, total elapsed time 00:00:01.5376710.
Inserted 10000 records into list, elapsed time: 00:00:01.7376893, total elapsed time 00:00:02.4404358.
Inserted 15000 records into list, elapsed time: 00:00:02.7716668, total elapsed time 00:00:03.4748621.
Inserted 20000 records into list, elapsed time: 00:00:03.9890341, total elapsed time 00:00:04.6926910.
Inserted 25000 records into list, elapsed time: 00:00:05.1933504, total elapsed time 00:00:05.8974158.
Inserted 30000 records into list, elapsed time: 00:00:06.6847656, total elapsed time 00:00:07.3892500.
Inserted 35000 records into list, elapsed time: 00:00:10.7670988, total elapsed time 00:00:11.4720142.
Inserted 40000 records into list, elapsed time: 00:00:28.0815546, total elapsed time 00:00:28.7868870.
Inserted 45000 records into list, elapsed time: 00:01:00.8765264, total elapsed time 00:01:01.5823218.
Inserted 50000 records into list, elapsed time: 00:01:49.0085502, total elapsed time 00:01:49.7147589.
Inserted 55000 records into list, elapsed time: 00:02:31.0452093, total elapsed time 00:02:31.7519024.
Inserted 60000 records into list, elapsed time: 00:03:20.4981843, total elapsed time 00:03:21.2052793.
Inserted 65000 records into list, elapsed time: 00:04:26.4099773, total elapsed time 00:04:27.1175721.
Inserted 70000 records into list, elapsed time: 00:05:27.9312605, total elapsed time 00:05:28.6392825.
Inserted 75000 records into list, elapsed time: 00:06:46.3439561, total elapsed time 00:06:47.0524274.
Inserted 80000 records into list, elapsed time: 00:08:14.2444098, total elapsed time 00:08:14.9533567.
Inserted 85000 records into list, elapsed time: 00:09:48.3997141, total elapsed time 00:09:49.1091194.
Inserted 90000 records into list, elapsed time: 00:11:29.8573649, total elapsed time 00:11:30.5671798.
Inserted 95000 records into list, elapsed time: 00:13:09.8709034, total elapsed time 00:13:10.5811210.
Inserted 100000 records into list, elapsed time: 00:15:16.4506298, total elapsed time 00:15:17.1613391.
Inserted 105000 records into list, elapsed time: 00:17:09.4155188, total elapsed time 00:17:10.1266124.
Inserted 110000 records into list, elapsed time: 00:19:27.7560402, total elapsed time 00:19:28.4675951.
Inserted 115000 records into list, elapsed time: 00:21:43.1077075, total elapsed time 00:21:43.8197227.
Inserted 120000 records into list, elapsed time: 00:23:54.8152649, total elapsed time 00:23:55.5276633.
Inserted 125000 records into list, elapsed time: 00:26:00.9206303, total elapsed time 00:26:01.6334805.
Inserted 130000 records into list, elapsed time: 00:27:59.3610854, total elapsed time 00:28:00.0743443.
Inserted 135000 records into list, elapsed time: 00:30:19.1292082, total elapsed time 00:30:19.8428843.
Inserted 140000 records into list, elapsed time: 00:32:50.3883024, total elapsed time 00:32:51.1023912.
Inserted 145000 records into list, elapsed time: 00:35:19.1582880, total elapsed time 00:35:19.8727696.
Inserted 150000 records into list, elapsed time: 00:38:06.5936198, total elapsed time 00:38:07.3084925.
Inserted 155000 records into list, elapsed time: 00:40:59.8120636, total elapsed time 00:41:00.5273436.
Inserted 160000 records into list, elapsed time: 00:43:26.8877093, total elapsed time 00:43:27.6034234.
Inserted 165000 records into list, elapsed time: 00:46:13.7403832, total elapsed time 00:46:14.4565184.
Inserted 170000 records into list, elapsed time: 00:49:17.1470592, total elapsed time 00:49:17.8636632.
Inserted 175000 records into list, elapsed time: 00:52:25.2611825, total elapsed time 00:52:25.9782587.
Inserted 180000 records into list, elapsed time: 00:55:14.3619783, total elapsed time 00:55:15.0794720.
Inserted 185000 records into list, elapsed time: 00:58:02.8347567, total elapsed time 00:58:03.5526672.
Inserted 190000 records into list, elapsed time: 01:01:04.9061189, total elapsed time 01:01:05.6244007.
Inserted 195000 records into list, elapsed time: 01:03:54.9140452, total elapsed time 01:03:55.6327840.
Inserted 200000 records into list, elapsed time: 01:06:52.2178177, total elapsed time 01:06:52.9369828.
Inserted 205000 records into list, elapsed time: 01:10:01.5741111, total elapsed time 01:10:02.2937813.
Inserted 210000 records into list, elapsed time: 01:13:31.9921117, total elapsed time 01:13:32.7121679.
Inserted 215000 records into list, elapsed time: 01:17:10.1325128, total elapsed time 01:17:10.8530487.
Inserted 220000 records into list, elapsed time: 01:20:44.3941916, total elapsed time 01:20:45.1151520.
Inserted 225000 records into list, elapsed time: 01:24:37.3879278, total elapsed time 01:24:38.1093512.
Inserted 230000 records into list, elapsed time: 01:28:06.1678372, total elapsed time 01:28:06.8897135.
Inserted 235000 records into list, elapsed time: 01:31:55.0756463, total elapsed time 01:31:55.7980041.
Inserted 240000 records into list, elapsed time: 01:35:32.8303907, total elapsed time 01:35:33.5531674.
Inserted 245000 records into list, elapsed time: 01:38:15.4338688, total elapsed time 01:38:16.1570603.
Inserted 250000 records into list, elapsed time: 01:41:03.9358269, total elapsed time 01:41:04.6594130.
Inserted 255000 records into list, elapsed time: 01:44:03.7310638, total elapsed time 01:44:04.4550818.
Inserted 260000 records into list, elapsed time: 01:47:27.4805956, total elapsed time 01:47:28.2050487.
Inserted 265000 records into list, elapsed time: 01:51:00.3374803, total elapsed time 01:51:01.0624061.
Inserted 270000 records into list, elapsed time: 01:54:06.2178364, total elapsed time 01:54:06.9431830.
I'm not sure how feasible this would be, but perhaps one possible optimization for bulk inserting documents would be to determine if the record count is greater than 40,000 and if it is, implement a system of sharding the data into separate temporary collections which could then be merged back into the intended target collection behind the scenes.  Something along the lines of:
//pseudo code
collection; //mycol
temporaryCollection ; //mycol_
temporaryCollection.Insert(docs); //up to 40,000 documents at 5,000 per insert
collection.Merge(temporaryCollection); //merge shard into target collection
db.Drop(temporaryCollection); //drop and continue for the next 40,000 documents

Just an idea :)

Here is an exert from one of the smaller JSON files:
{
    "id":5,
    "name":"Reilly Hub",
    "system_id":396,
    "updated_at":1488926146,
    "max_landing_pad_size":"L",
    "distance_to_star":171,
    "government_id":64,
    "government":"Corporate",
    "allegiance_id":3,
    "allegiance":"Federation",
    "state_id":80,
    "state":"None",
    "type_id":8,
    "type":"Orbis Starport",
    "has_blackmarket":false,
    "has_market":true,
    "has_refuel":true,
    "has_repair":true,
    "has_rearm":true,
    "has_outfitting":true,
    "has_shipyard":true,
    "has_docking":true,
    "has_commodities":true,
    "import_commodities":[
        "Pesticides",
        "Aquaponic Systems",
        "Biowaste"
    ],
    "export_commodities":[
        "Mineral Oil",
        "Fruit and Vegetables",
        "Grain"
    ],
    "prohibited_commodities":[
        "Narcotics",
        "Tobacco",
        "Combat Stabilisers",
        "Imperial Slaves",
        "Slaves",
        "Personal Weapons",
        "Battle Weapons",
        "Toxic Waste",
        "Wuthielo Ku Froth",
        "Bootleg Liquor",
        "Landmines"
    ],
    "economies":[
        "Agriculture"
    ],
    "shipyard_updated_at":1489126071,
    "outfitting_updated_at":1489126066,
    "market_updated_at":1489126065,
    "is_planetary":false,
    "selling_ships":[
        "Adder",
        "Eagle Mk. II",
        "Hauler",
        "Sidewinder Mk.   Please let me know if there's anything I can do to assist.

One last thing I'm not quite sure of.  When you said "try close/open datafile in each batch insert (could be page cache memory consume)." Are you referring to opening and closing the LiteDB data file, or the reader of the JSON file?  Thanks.

Also just a heads up on the machine I'm testing with.  I'm using a 12 core 4.2 GHz machine with 128GB RAM and 1TB M.2 PCI-E drive.  So machine bottlenecks shouldn't be an issue here.

Looks like you got a database for the pan-galaxy?

@godlikemouse, try this:
using (JsonStream stream = new JsonStream(File.OpenRead(path)))
{
    string importName = string.Format("{0}-import", collectionName);

    using (var db = new LiteEngine("filename=test.db; journal=false"))
    {
        db.DropCollection(importName, docs.Select(x => x.AsDocument));
    }

    List<BsonValue> docs = new List<BsonValue>();
    string json = null;
    while ((json = stream.ReadLine()) != null)
    {

        try
        {

            docs.Add(JsonSerializer.Deserialize(json));

            if (++count > 0 && count % BULK_LIMIT == 0)
            {

                sw.Start();

                using (var db = new LiteEngine("filename=test.db; journal=false"))
                {
                    db.Insert(importName, docs.Select(x => x.AsDocument));
                }

                sw.Stop();
                Debug.WriteLine("Inserted {0} records : {1} total, elapsed time: {2}, total elapsed time {3}.", BULK_LIMIT, count, sw.Elapsed, csw.Elapsed);
                sw.Reset();
                docs.Clear();
            }
        }
        catch(Exception ex)
        {
            Console.Error.WriteLine(ex.Message);
        }
    }

    if (docs.Count > 0)
    {
        sw.Start();
        using (var db = new LiteEngine("filename=test.db; journal=false"))
        {
            db.Insert(importName, docs.Select(x => x.AsDocument));
        }
        sw.Stop();
        Debug.WriteLine("Inserted {0} records : {1} total, elapsed time: {2}, total elapsed time {3}.", BULK_LIMIT, count, sw.Elapsed, csw.Elapsed);
    }
}

@xied75 It's actually from the Elite Dangerous game.
@mbdavid I've implemented the above code but changed:
db.DropCollection(importName, docs.Select(x => x.AsDocument));
to
db.DropCollection(importName);
No change result wise.  There is a dramatic increase in insert time starting around 40K records.
Inserted 5000 records : 5000 total, elapsed time: 00:00:00.7832291, total elapsed time 00:00:01.7742849.
Inserted 5000 records : 10000 total, elapsed time: 00:00:01.1015218, total elapsed time 00:00:03.6003434.
Inserted 5000 records : 15000 total, elapsed time: 00:00:01.0323648, total elapsed time 00:00:05.6693271.
Inserted 5000 records : 20000 total, elapsed time: 00:00:01.3664129, total elapsed time 00:00:07.8111386.
Inserted 5000 records : 25000 total, elapsed time: 00:00:01.1922187, total elapsed time 00:00:09.8823094.
Inserted 5000 records : 30000 total, elapsed time: 00:00:01.3958514, total elapsed time 00:00:11.9560483.
Inserted 5000 records : 35000 total, elapsed time: 00:00:01.8160972, total elapsed time 00:00:14.7466903.
Inserted 5000 records : 40000 total, elapsed time: 00:00:04.4946808, total elapsed time 00:00:20.0723357.
Inserted 5000 records : 45000 total, elapsed time: 00:00:46.3005596, total elapsed time 00:01:07.3992580.
Inserted 5000 records : 50000 total, elapsed time: 00:00:29.9726260, total elapsed time 00:01:38.1727240.
Inserted 5000 records : 55000 total, elapsed time: 00:01:13.8675638, total elapsed time 00:02:53.0437222.
Inserted 5000 records : 60000 total, elapsed time: 00:01:04.2471502, total elapsed time 00:03:58.1291849.
Inserted 5000 records : 64495 total, elapsed time: 00:01:37.6663914, total elapsed time 00:05:36.3745306.

If I had to guess, I would say that the seek to end paging during insert may need a bit of optimization for bulk inserts.  I think what we're seeing is after a bulk insert of 5000 records, the wait time is the time finding the next open page after the previous last record.  Perhaps keeping the pointer to the last insert area during a bulk operation or sharding would help speed up the inserts.  Please let me know if there is anything else I provide or help with. Thanks.

Hi @godlikemouse, i will investigate during this weekend, but thats not reason for that. LiteDB uses skip list with 32-level. Skip lists works with O^n, so to insert 64kb it´s not more than 16 step (pages) to get index position and it´s depend if datafile is open or need open again. Problem must about caching pages or something like that. I will keep you informed

Hi @mbdavid , thanks for the update and the assistance.  Before I saw this message I downloaded the source code and began setting stopwatches in and around insert methods.  I've traced the issue down to the PageServce.AddToFreeList method so far.  I'm not sure specifically where the bottleneck is but I thought I'd let you know what I've found so far (you probably already knew this though) :)

Hi @mbdavid , I think you're right.  I'm seeing now that the GetPage(nextPageID) appears to be growing in retrieval time as the inserts occur.
first 5000: 00:00:00.0000232 max
next 5000: 00:00:00.0001086 max
next 5000: 00:00:00.0080962 max
...
last 5000: 00:00:00.1492187 max
The above is simply tracking the maximum amount of time the GetPage() call takes per bulk insert.

humm, this is new to me. Maybe problem are in AddToFreeList method. But what is more strange that if you open/close datafile there is no problem on this. Just to know, AddToFreeList (that uses GetPage) like avaiable pages in a continuous list. And I know a possible bug in my implementation (that came from first version). If your used page remains bytes enouth for more data, you your next data is always bigger than this size, you will always get bigger this list (and this list is O(n) complexity). If it's the case will be a great case to try fixed this :)

Great!  Please let me know if I can be of any assistance in providing a data set or testing for you.  Thanks again.

oh my... i got it!!! :) I changed this to test if was possible reduce datafile size and forgot to have after!
https://github.com/mbdavid/LiteDB/blob/master/LiteDB/Engine/Pages/DataPage.cs#L20
I will revert this and try

Awesome 👍 I can't wait to check it out.  Thanks.

That looks like that may have been it, tested using the stations.json, got the following:
Inserted 5000 records : 5000 total, elapsed time: 00:00:01.1685254, total elapsed time 00:00:03.9108816.
Inserted 5000 records : 10000 total, elapsed time: 00:00:00.8184333, total elapsed time 00:00:05.9691680.
Inserted 5000 records : 15000 total, elapsed time: 00:00:01.2047131, total elapsed time 00:00:08.7821987.
Inserted 5000 records : 20000 total, elapsed time: 00:00:00.9522625, total elapsed time 00:00:10.9204821.
Inserted 5000 records : 25000 total, elapsed time: 00:00:00.9803163, total elapsed time 00:00:13.3693458.
Inserted 5000 records : 30000 total, elapsed time: 00:00:00.8960723, total elapsed time 00:00:15.3639652.
Inserted 5000 records : 35000 total, elapsed time: 00:00:01.1181613, total elapsed time 00:00:18.2523672.
Inserted 5000 records : 40000 total, elapsed time: 00:00:00.9317909, total elapsed time 00:00:20.5701199.
Inserted 5000 records : 45000 total, elapsed time: 00:00:01.1261127, total elapsed time 00:00:23.2997889.
Inserted 5000 records : 50000 total, elapsed time: 00:00:00.9147380, total elapsed time 00:00:25.5833090.
Inserted 5000 records : 55000 total, elapsed time: 00:00:01.2603071, total elapsed time 00:00:28.4274433.
Inserted 5000 records : 60000 total, elapsed time: 00:00:01.3036738, total elapsed time 00:00:31.0259107.
Inserted 5000 records : 64495 total, elapsed time: 00:00:01.1868521, total elapsed time 00:00:33.3413561.

Next I'll test using the large data set.  Thanks again for the work so far.

Great work @mbdavid , this fixed the issue here.  Please feel free to close this.

Hi @godlikemouse, added InsertBulk using same idea here

Im Trying to read a json file and inserting into lite DB
openLoadDef.FileName = "";
openLoadDef.ShowDialog();
string errordef = openLoadDef.FileName;
using (var sr = new StreamReader(errordef))
{
var docs = JsonSerializer.DeserializeArray(sr);
}
Json file content: { "name":"John", "age":31, "city":"New York" }
I'm getting the following error
Unexpected JSON token: name
Stack Trace:    at LiteDB.JsonToken.Expect(JsonTokenType type)
at LiteDB.JsonReader.d__5.MoveNext()
at System.Linq.SystemCore_EnumerableDebugView`1.get_Items()
I Tried to debug the issue .. but unable to do so.. kindly help.

@vijaykirann People normally is not very fast on responding to a closed issue. Also your issue might not share anything with this closed one, so better create a new issue. And your code fragment is too short to figure out what's going on.

thanks @xied75 .. I have opened a new issue.however I was trying read a jsonfile and inserting into litedb.. however the above mentioned code snippets arent working for me ..


[nfc] [ownership] move ownership checks to visitors in sil combine

I'm going to update one visitor at a time to support ownership. This patch updates all the visitors to bail on ownership (except for the non-ossa instructions).

@swift-ci please smoke test.

Can you put an assert that we are not visiting something with ownership in visitReleaseValueInst? visitRetainValueInst? visitStrongRetainInst, visitStrongReleaseInst. This is more of a documentation that the code is expecting non-ossa code.
In a forthcoming commit, can you get rid of those simple nullptr visitors as well?
After that LGTM.

@gottesmm updated. I'll make a patch for the nullptr fix.

@swift-ci please smoke test and merge.

@swift-ci please smoke test and merge


Update image requirement from 0.24 to 0.25

Objective

Closes https://github.com/bevyengine/bevy/pull/12415

Solution

Refactored code that was changed/deprecated in image 0.25.
Please review this PR carefully since I'm just making the changes without any context or deep knowledge of the module.

I think there are a few issues here:

read_image needs a buffer sized to total_bytes (not just with that capacity)
The code that was adding alpha back into the image data is still needed if we are returning a TextureFormat::Rgba32Float. The hdr format never has an alpha channel.

There's an .hdr file in the assets folder you can test with -- modify examples/2d/sprite.rs to load textures/spiaggia_di_mondello_1k.hdr.

Thanks a lot for the review, I needed some guidance here. I'll take a look at these issues and push some updates later.

@rparrett I managed to make changes so the example you've provided would load that HDR image properly. Please check if what I'm doing now is correct; I'm still unsure if I'm sizing the data buffer properly.
Also, I'd appreciate some input on the error messages since I'm not convinced on how I wrote them.


Update Rust to 1.75.0

Release notes.

Tag workflow as per CONTRIBUTING.md and per https://github.com/neondatabase/neon/pull/6222#issuecomment-1866669516 : https://github.com/neondatabase/neon/actions/runs/7446926961

Note for next upgrade: add the run-extra-build-macos label to it, like in #6294, to ensure there is no macOS regression.


undefined: SQLiteConn

Error when building with golang:1.12-alpine
CGO_ENABLED=0  go build

error
# github.com/mattn/go-sqlite3
../github.com/mattn/go-sqlite3/sqlite3_opt_preupdate.go:12:16: undefined: SQLiteConn

Why are you building with cgo disabled?

Because when doing unit testing, I hope to use golang-alpine to build the image directly, because it will be smaller.
 Using alpine does not necessitate disabling cgo.

I just found this sentence in the README.md document, please close this issue.
Important: because this is a CGO enabled package you are required to set the environment variable CGO_ENABLED=1 and have a gcc compile present within your path.

So, for what it's worth, I started getting this error message when building with Go 1.12 on linux-amd64 (RHEL 7.4) for non-native arches from 2ea5857c0eb3179b078ce5fb881ace11eb28f285 onwards -- I'm guessing from the README I now might need a cross-compiling CC to be set, where I didn't seem to have to worry about that before.
Not really a maintainer's problem, I know, but I thought it might be useful to report.

@ikirker This library certainly cannot be compiled without cgo in any meaningful sense. And using cgo has always required you to explicitly set CGO_ENABLED=1 and CC=<cross compiler> when cross-compiling. I can't speak to how it managed to work for you previously, but it probably should not have.

Sometimes we may have multiple environments, such as staging and qa and prd, staging using sqlite, qa and prd using mysql
In the qa and prd environment, we want to use CGO_ENBALE=0 build, but because staging environment uses sqlite, it will inevitably use _ "github.com/mattn/go-sqlite3" in our code, which will lead to build failure

@relax-admin That sounds like a bad idea. You should be using a consistent setup in each environment. http://michael.robellard.com/2015/07/dont-test-with-sqllite-when-you-use.html

@rittneje Thank you very much for this article, and now our test and build environment is unified to mysql.

G


Made WireGPU_matScreen a translucent material.

Change Notes:
Added support to GPUlib for transparent screens.
Added a translucent option to the wire monitor functions.
Changed render group for screen entities to support translucent
monitors.
Changed EGP emitters to use render targets.
Information
So recently I learned how to make render targets translucent, with that in mind I thought wire could do with the ability to draw translucent screens. So I added support for it, changing as little code as possible.
whilst wire will now support translucent screens I have not been able to figure out how to get a few of the screen entities to be translucent (e.g console screens). somebody else who understands those entities better will need to do this.
I have also changed how EGP emitters draw, so they now have a 512 by 512 area to draw too and will be clipped like originally planned. This will probably restrict some of the things players have previous created that render past this boundary.
I don't quite understand how to properly define wire monitors but I have gotten away with adding 1 new monitor witch I have named "Window", though I am sure somebody will go crazy adding new transparent monitors.
Please be aware that a few wire developers should test this PR first since I have not had time to test all the things, that said I have not managed to break anything whilst testing. the only thing I am concerned about is the GPU switcher tool, this is because I don't quite understand it well enough to mess with it. If this tool is broken however I will be happy to speak to a wire dev and try to resolve the issue.
I would like to say thank you to TankNut for helping me with the project that taught me how to make this possible.
Live long and prosper friends.

haven't tested yet but looks good to me

As was pointed out on discord, this would remove the ability to use certain materials on emitters, because those materials can't be drawn on rendertargets. I'm pretty sure there is a way to make all materials work on rendertargets because I have a faint memory of someone getting it to work in the past, but I have no clue how.

The issue @Divran mentioned has now been fixed. I have changed the GPU lib again to allow png files to be rendered.

Done some testing and everything works fine except pngs which don't work at all for me. As soon as those are fixed I say merge

PNG files work for me, I have tested them extensively.

Applying a material to the screen seems to just make it completely invisible for me.
http://www.wizry.net/video/other/egp.mp4

We have determined that this is down to MSAA being turned on, we are currently looking into disabling smoothing on the GPU material, doesn't look like we will get anywhere how ever.

https://github.com/Facepunch/garrysmod-issues/issues/3664


Requested by Justin Cowart.

Done using existing RGB mutliples


[FEATURE] Make generators interactive

Description
I would like to add a combination of transform : scale,  translate and hover class to these generators to make them look interactive.

Screenshots

It's great having you contribute to this project by creating an issue Make sure you are assigned to this before you work on it and you read the Contributing.md file, Thank you! Welcome to the community :nerd_face:

Can someone assign this to me?

Thanks for creating this issue, but it's not needed


bounds check for last_interrupt_index

Based on discussion from #316 I'm proposing this pull to add bounds checking for the last_interrupt_index to all functions in the IRQManager class that add interrupts.

@iabdalkader   I mean to @ you in the description and forgot.  Does this look ok?

@iabdalkader I mean to @ you in the description and forgot. Does this look ok?

Thanks @delta-G for sending this. This looks good, but I need to take a closer look on Monday.


Create update [v0.5.1a] (for 1.18.2) broke the mod

Latest Create update [v0.5.1a] (for 1.18.2) broke this mod. 1 error during loading
Create Ore Excavation (createoreexcavation) has failed to load correctly
java.lang.NoClassDefFoundError: com/simibubi/create/foundation/block/BlockStressValues$IStressValueProvider

Fixed in the latest update


installation issue - 3rd party reference

When cloning the repository:
git clone --recursive https://github.com/damellis/ESP.git
it fails with:
fatal: clone of 'git@github.com:braitsch/ofxSmartFont.git' into submodule path 'modules/ofxSmartFont' failed
Failed to recurse into submodule path 'third-party/ofxParagraph'

Probably due to a change in git reference for the 3rd part module.
However as a workaround the missing module can be cloned manually from:
https://github.com/braitsch/ofxSmartFont.git

Mmm, that comes from ofxParagraph repository: https://github.com/braitsch/ofxParagraph/blob/master/.gitmodules#L3
To use git clone git@github.com:XXXX, one needs to setup the ssh key properly. Otherwise, the workaround you mentioned can work.
Closing this as not a fix-able issue from our side.


Remove direct mgmt dependencies from azure-sdk-tools. Lock the rest.

See title.
[https://github.com/Azure/azure-sdk-for-python/blob/30d108fa82e21dd38cb079348f7a316d51cc1929/tools/azure-sdk-tools/setup.py#L7](What needs to be locked.)
Wholesale remove any direct deps on mgmt packages in the above list. Ensure this works with all (min/latest) pipelines and at least regression test against core.
This is part of the effort to clean up our tools/ directory. The other half will be figuring out what we do with azure-devtools. Either merge it into azure-sdk-tools or move it into the external repo consumed by the CLI team.

This is not feasible. Going to tackle this from a different approach.


Age (date) validation

What think about a validation for age? It's appropriated?
<input ... validation="age:18|required" />

Related #130

just use the min_num:18 with the alt property Wiki - Alternate Text
validation="min_num:18:alt=Must be legal of age"

Really? I need read again the docs! Sorry for my mistake.

@ghiscoding I mean for birthdate validation, not for a integer number.

Do you mean date calculation? That's getting too customized, you should look into building your own custom javascript function for that... Wiki - Custom Validation

Yep, is a good alternative. Thanks!


Broken installation script

Apparently there has been a change on the default branch which has broken the installation script.
This line https://github.com/spf13/spf13-vim/blob/master/bootstrap.sh#L20 clones the repo without a branch argument, which clones the default (3.0) that does not have the .vim folder.
I'll be providing a PR soon.

@spf13 quick but fatal bug here.

@LuisAlejandro : the last commit to master was in 2014. using 3.0 is intended. If there's a bug, it isn't that.

Well, i guess it was a false positive, somehow the URL shortener redirected me to:
https://github.com/spf13/spf13-vim/blob/master/bootstrap.sh
instead of:
https://github.com/spf13/spf13-vim/blob/3.0/bootstrap.sh
Which will fail because of what i explained. You should change the script anyway because someone installing from master will get a failed installation, don't you think?

change it to ? master? no.

I've contacted @spf13 to see if he can look into the url shortener issue you're seeing.

@jrobeson
Change it to master in master? makes no sense?
:-(

i'd rather it fail completely than allow people to install the master branch. seems like it might be better to adapt the installer to just error out on fresh installs. It could say something like "You're installing an unmaintained version of spf-13, please check out 3.0 branch and run the bootstrap script

Closing this, as apparently not a bug.


Resetting available status options after rerender

Critical Changes
Changes
Issues Closed

@davidhabib I realized after you merged my previous PR that the status options were resetting after rerender. Updating to refilter after rerender.


Remove the PP/ prefix from some of the RefNos in the born-digital space in the storage service

Use the new deletion script + reupload the bag.
Coordinate with Steph about what the IDs should be.

Closing as a dupe of #4923.


Feature request

geemap version: 0.6.13
Python version: 3.8.2
Operating System: Win10 64bit

Description
When i use geemap in Jupyterlab, the Map function can not output the map, instead of
Map(center=[40, -100], controls=(WidgetControl(options=['position'], widget=HBox(children=(ToggleButton(value=…

So, is there possible to let geemap support Jupyterlab development?
What i do
Codes 1:
import geemap
Map = geemap.Map(center=[40,-100], zoom=4)
Map

Output 1:
Map(center=[40, -100], controls=(WidgetControl(options=['position'], widget=HBox(children=(ToggleButton(value=…

Codes 2:
import geemap.eefolium as emap
Map = emap.Map(center=[40,-100], zoom=4)
Map

Output 2: Normal map.
Feature improve
Supporting ipyleaflet under Jupyterlab with geemap.

Problem resolved, ref: https://github.com/jupyter-widgets/ipyleaflet
jupyter labextension install @jupyter-widgets/jupyterlab-manager jupyter-leaflet


Tabs are not functioning properly for multiple tables on same page.

It can be a use case for the dashboard pages or overview pages where we show multiple tables and wants to have tabs in different tables on the same page.
Ex: I tried to update the Finance dashboard, we want to show the invoice table and project awaiting approval table both with tabs.
But when we load the dashboard page only one table is active by default or only one tab is active among all the tabs on the page.
Current behaviour:

Expected behaviour:

@frjo @theskumar I have created a draft PR(#3115) with the changes that can help you to populate the above scenario. Just pull the branch and look into the finance1 dashboard.

Looks like the javascript in hypha/static_src/src/javascript/apply/tabs.js needs to be updated to handle multiple tab sections. Right now it thinks all the tabs is in one single group.

Record: @theskumar has already fixed this issue with alpine in PR #3115.


Refactor lock LLVM IR generation code

Create a struct containing locking info and have a pointer in the class
class Foo{
  struct *lockingInfo

}

struct{
    bool isLockBeingUpdated;
    pthread_t *ownerThread;
    int writerCount;
    int readerCount;
}

And then we refactor so calling lock / unlock will call a function rather than generating IR for each instance of lock/unlock.
_lock(struct x, locktype){

if(lock is being updated){
  spin;
}
currThreadOwnsLock = check current pthread with the lock's pthread
noWritersPresentOnOtherThreads = numWriters ==0 || currThreadOwnsLock

 if( type == Reader){
       canAcquireLock = noWritersPresentOnOtherThreads;
   }
else{ // mode is writer
   canAcquireLock = noWritersPresentOnOtherThreads && noReadersPresent;
}

CAS( lockUpdated, oldval=0, newVal=1);
// if fail then loop to lock update

// now we're in critical section to update lock info (meta lock within a lock)
if reader{
   readerCount++
}

if writer {
update owning thread to this thread
writerCount++
}

mem fence

store lockUpdated -> 0 // release update lock
}

And for the unlock operation
unlock( struct, lock type){

    spin on LockUpdate mutex

    if (reader lock){
           readerCount--
      }

     if writer {
             writerCount--
    }
mem fence

store lockUpdated -> 0 // release update lock

}

(Bug - not updating field)
let (i = 0; i< 10; i++){
x.f := x.f + i
}

x.f is set to 0 if using a locked capability, rather than 55 (1+2+...+10).

Just write the lock in C and link in the function, and use C atomics / insert memory fence using LLVM.

Right now the lock is taken on the whole object, we should instead have a lock per capability, covering only those fields touched by the capability.


Fix consul monitoring test flakes

Previously, the tests would wait for some period of time and check if it
got any updates. This timing sensitive test fails often when CPU is
throttled or slow, especially in -race or coverage mode. Additionally,
one of the tests depended on the order of a map which is undefined.
This change sorts the map output to be deterministic and changes the
test to poll for success rather than wait and check.
The failures are pretty easily reproducible with docker run -v "${PWD}":"${PWD}" -w "${PWD}" -e GOPATH=$GOPATH --cpus=1 golang:1.12.7 go test ./pilot/pkg/serviceregistry/consul -run TestController -count 10 -race, and no longer occur for me after this
Fixes https://github.com/istio/istio/issues/15820
[ ] Configuration Infrastructure
[ ] Docs
[ ] Installation
[x] Networking
[ ] Performance and Scalability
[ ] Policies and Telemetry
[ ] Security
[x] Test and Release
[ ] User Experience
[ ] Developer Infrastructure

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: howardjohn
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

pilot/OWNERS [howardjohn]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@howardjohn: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

e2e-dashboard-master
224098e6f4d3edbd1c93f2621f5c7487ffd4b48d
link
/test e2e-dashboard-master

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@howardjohn: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

e2e-dashboard-master
224098e6f4d3edbd1c93f2621f5c7487ffd4b48d
link
/test e2e-dashboard-master

e2e-bookInfoTests-envoyv2-v1alpha3-master
224098e6f4d3edbd1c93f2621f5c7487ffd4b48d
link
/test e2e-bookInfoTests-envoyv2-v1alpha3-master

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@howardjohn: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

e2e-dashboard-master
224098e6f4d3edbd1c93f2621f5c7487ffd4b48d
link
/test e2e-dashboard-master

e2e-bookInfoTests-envoyv2-v1alpha3-master
224098e6f4d3edbd1c93f2621f5c7487ffd4b48d
link
/test e2e-bookInfoTests-envoyv2-v1alpha3-master

istio-pilot-multicluster-e2e-master
224098e6f4d3edbd1c93f2621f5c7487ffd4b48d
link
/test istio-pilot-multicluster-e2e-master

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@howardjohn: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

e2e-dashboard-master
224098e6f4d3edbd1c93f2621f5c7487ffd4b48d
link
/test e2e-dashboard-master

e2e-bookInfoTests-envoyv2-v1alpha3-master
224098e6f4d3edbd1c93f2621f5c7487ffd4b48d
link
/test e2e-bookInfoTests-envoyv2-v1alpha3-master

istio-pilot-multicluster-e2e-master
224098e6f4d3edbd1c93f2621f5c7487ffd4b48d
link
/test istio-pilot-multicluster-e2e-master

e2e-mixer-no_auth-master
224098e6f4d3edbd1c93f2621f5c7487ffd4b48d
link
/test e2e-mixer-no_auth-master

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@howardjohn: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

e2e-dashboard-master
224098e6f4d3edbd1c93f2621f5c7487ffd4b48d
link
/test e2e-dashboard-master

e2e-bookInfoTests-envoyv2-v1alpha3-master
224098e6f4d3edbd1c93f2621f5c7487ffd4b48d
link
/test e2e-bookInfoTests-envoyv2-v1alpha3-master

istio-pilot-multicluster-e2e-master
224098e6f4d3edbd1c93f2621f5c7487ffd4b48d
link
/test istio-pilot-multicluster-e2e-master

e2e-mixer-no_auth-master
224098e6f4d3edbd1c93f2621f5c7487ffd4b48d
link
/test e2e-mixer-no_auth-master

istio_auth_sds_e2e-master
224098e6f4d3edbd1c93f2621f5c7487ffd4b48d
link
/test istio_auth_sds_e2e-master

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@howardjohn: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

e2e-dashboard-master
224098e6f4d3edbd1c93f2621f5c7487ffd4b48d
link
/test e2e-dashboard-master

e2e-bookInfoTests-envoyv2-v1alpha3-master
224098e6f4d3edbd1c93f2621f5c7487ffd4b48d
link
/test e2e-bookInfoTests-envoyv2-v1alpha3-master

istio-pilot-multicluster-e2e-master
224098e6f4d3edbd1c93f2621f5c7487ffd4b48d
link
/test istio-pilot-multicluster-e2e-master

e2e-mixer-no_auth-master
224098e6f4d3edbd1c93f2621f5c7487ffd4b48d
link
/test e2e-mixer-no_auth-master

istio_auth_sds_e2e-master
224098e6f4d3edbd1c93f2621f5c7487ffd4b48d
link
/test istio_auth_sds_e2e-master

e2e-simpleTests-master
224098e6f4d3edbd1c93f2621f5c7487ffd4b48d
link
/test e2e-simpleTests-master

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@howardjohn: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

e2e-dashboard-master
224098e6f4d3edbd1c93f2621f5c7487ffd4b48d
link
/test e2e-dashboard-master

e2e-bookInfoTests-envoyv2-v1alpha3-master
224098e6f4d3edbd1c93f2621f5c7487ffd4b48d
link
/test e2e-bookInfoTests-envoyv2-v1alpha3-master

istio-pilot-multicluster-e2e-master
224098e6f4d3edbd1c93f2621f5c7487ffd4b48d
link
/test istio-pilot-multicluster-e2e-master

e2e-mixer-no_auth-master
224098e6f4d3edbd1c93f2621f5c7487ffd4b48d
link
/test e2e-mixer-no_auth-master

istio_auth_sds_e2e-master
224098e6f4d3edbd1c93f2621f5c7487ffd4b48d
link
/test istio_auth_sds_e2e-master

e2e-simpleTests-master
224098e6f4d3edbd1c93f2621f5c7487ffd4b48d
link
/test e2e-simpleTests-master

istio-pilot-e2e-envoyv2-v1alpha3-master
224098e6f4d3edbd1c93f2621f5c7487ffd4b48d
link
/test istio-pilot-e2e-envoyv2-v1alpha3-master

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@howardjohn: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

e2e-dashboard-master
224098e6f4d3edbd1c93f2621f5c7487ffd4b48d
link
/test e2e-dashboard-master

e2e-bookInfoTests-envoyv2-v1alpha3-master
224098e6f4d3edbd1c93f2621f5c7487ffd4b48d
link
/test e2e-bookInfoTests-envoyv2-v1alpha3-master

istio-pilot-multicluster-e2e-master
224098e6f4d3edbd1c93f2621f5c7487ffd4b48d
link
/test istio-pilot-multicluster-e2e-master

e2e-mixer-no_auth-master
224098e6f4d3edbd1c93f2621f5c7487ffd4b48d
link
/test e2e-mixer-no_auth-master

istio_auth_sds_e2e-master
224098e6f4d3edbd1c93f2621f5c7487ffd4b48d
link
/test istio_auth_sds_e2e-master

e2e-simpleTests-master
224098e6f4d3edbd1c93f2621f5c7487ffd4b48d
link
/test e2e-simpleTests-master

istio-pilot-e2e-envoyv2-v1alpha3-master
224098e6f4d3edbd1c93f2621f5c7487ffd4b48d
link
/test istio-pilot-e2e-envoyv2-v1alpha3-master

e2e-simpleTests-distroless-master
224098e6f4d3edbd1c93f2621f5c7487ffd4b48d
link
/test e2e-simpleTests-distroless-master

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@howardjohn: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

e2e-dashboard-master
224098e6f4d3edbd1c93f2621f5c7487ffd4b48d
link
/test e2e-dashboard-master

e2e-bookInfoTests-envoyv2-v1alpha3-master
224098e6f4d3edbd1c93f2621f5c7487ffd4b48d
link
/test e2e-bookInfoTests-envoyv2-v1alpha3-master

istio-pilot-multicluster-e2e-master
224098e6f4d3edbd1c93f2621f5c7487ffd4b48d
link
/test istio-pilot-multicluster-e2e-master

e2e-mixer-no_auth-master
224098e6f4d3edbd1c93f2621f5c7487ffd4b48d
link
/test e2e-mixer-no_auth-master

istio_auth_sds_e2e-master
224098e6f4d3edbd1c93f2621f5c7487ffd4b48d
link
/test istio_auth_sds_e2e-master

e2e-simpleTests-master
224098e6f4d3edbd1c93f2621f5c7487ffd4b48d
link
/test e2e-simpleTests-master

istio-pilot-e2e-envoyv2-v1alpha3-master
224098e6f4d3edbd1c93f2621f5c7487ffd4b48d
link
/test istio-pilot-e2e-envoyv2-v1alpha3-master

e2e-simpleTests-distroless-master
224098e6f4d3edbd1c93f2621f5c7487ffd4b48d
link
/test e2e-simpleTests-distroless-master

e2e-bookInfoTests-trustdomain-master
224098e6f4d3edbd1c93f2621f5c7487ffd4b48d
link
/test e2e-bookInfoTests-trustdomain-master

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@howardjohn: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

e2e-dashboard-master
224098e6f4d3edbd1c93f2621f5c7487ffd4b48d
link
/test e2e-dashboard-master

e2e-bookInfoTests-envoyv2-v1alpha3-master
224098e6f4d3edbd1c93f2621f5c7487ffd4b48d
link
/test e2e-bookInfoTests-envoyv2-v1alpha3-master

istio-pilot-multicluster-e2e-master
224098e6f4d3edbd1c93f2621f5c7487ffd4b48d
link
/test istio-pilot-multicluster-e2e-master

e2e-mixer-no_auth-master
224098e6f4d3edbd1c93f2621f5c7487ffd4b48d
link
/test e2e-mixer-no_auth-master

istio_auth_sds_e2e-master
224098e6f4d3edbd1c93f2621f5c7487ffd4b48d
link
/test istio_auth_sds_e2e-master

e2e-simpleTests-master
224098e6f4d3edbd1c93f2621f5c7487ffd4b48d
link
/test e2e-simpleTests-master

istio-pilot-e2e-envoyv2-v1alpha3-master
224098e6f4d3edbd1c93f2621f5c7487ffd4b48d
link
/test istio-pilot-e2e-envoyv2-v1alpha3-master

e2e-simpleTests-distroless-master
224098e6f4d3edbd1c93f2621f5c7487ffd4b48d
link
/test e2e-simpleTests-distroless-master

e2e-bookInfoTests-trustdomain-master
224098e6f4d3edbd1c93f2621f5c7487ffd4b48d
link
/test e2e-bookInfoTests-trustdomain-master

istio-unit-tests-master
224098e6f4d3edbd1c93f2621f5c7487ffd4b48d
link
/test istio-unit-tests-master

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@howardjohn: The following tests failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

e2e-dashboard-master
224098e6f4d3edbd1c93f2621f5c7487ffd4b48d
link
/test e2e-dashboard-master

e2e-bookInfoTests-envoyv2-v1alpha3-master
224098e6f4d3edbd1c93f2621f5c7487ffd4b48d
link
/test e2e-bookInfoTests-envoyv2-v1alpha3-master

istio-pilot-multicluster-e2e-master
224098e6f4d3edbd1c93f2621f5c7487ffd4b48d
link
/test istio-pilot-multicluster-e2e-master

e2e-mixer-no_auth-master
224098e6f4d3edbd1c93f2621f5c7487ffd4b48d
link
/test e2e-mixer-no_auth-master

istio_auth_sds_e2e-master
224098e6f4d3edbd1c93f2621f5c7487ffd4b48d
link
/test istio_auth_sds_e2e-master

e2e-simpleTests-master
224098e6f4d3edbd1c93f2621f5c7487ffd4b48d
link
/test e2e-simpleTests-master

istio-pilot-e2e-envoyv2-v1alpha3-master
224098e6f4d3edbd1c93f2621f5c7487ffd4b48d
link
/test istio-pilot-e2e-envoyv2-v1alpha3-master

e2e-simpleTests-distroless-master
224098e6f4d3edbd1c93f2621f5c7487ffd4b48d
link
/test e2e-simpleTests-distroless-master

e2e-bookInfoTests-trustdomain-master
224098e6f4d3edbd1c93f2621f5c7487ffd4b48d
link
/test e2e-bookInfoTests-trustdomain-master

istio-unit-tests-master
224098e6f4d3edbd1c93f2621f5c7487ffd4b48d
link
/test istio-unit-tests-master

integ-new-install-k8s-presubmit-tests-master
224098e6f4d3edbd1c93f2621f5c7487ffd4b48d
link
/test integ-new-install-k8s-presubmit-tests-master

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/retest

/assign @nmittler

/lgtm


Graph Plots

Thanks for awesome lib,
Does Wisp comes with graph visualization? like those in d3 ?

Hey Omid, unfortunately today we only support canonical 2D charts, and no graph/network visualization (but would love to in the future - perhaps through evolving vega support)

Hey Omid, unfortunately today we only support canonical 2D charts, and no graph/network visualization (but would love to in the future - perhaps through evolving vega support)

Yes, that would be a good idea, I know that "sparkle project" also using vega. I tried to look at your code, it's very dependent on Highchart, that would be nice if you can provide me with some structure information about the project, then I may find some time to merge vega with wisp.

Yes - both the Highcharts implementation and repl API are highcharts focused.
We originally tried to sketch the type system around using other interfaces. You can see the work I started on vega here: https://github.com/quantifind/wisp/tree/austin/vega
Right now by implementing plot you can use any library - but you don't get support for things like line, column, bar, etc. - as those are highcharts specific methods. (for example, a graph implementation may not support them at all)
We can follow up by email if you'd like to discuss further - austin [at] name-of-the-company.com


Fix duplicate close file when deconstructing ZipArchive

Fixes #49012

Thanks!

Cherry-picked for 3.4.

Cherry-picked for 3.3.2.


A little help to adapt the code

Hi again.
So my LLM returns something like
"""comment"""
import funsearch

import numpy as np
import math

def priority_v0(el: int, w: int) -> float:
   return 0.0

def priority_v1(el: int, w: int) -> float:
  #Improved version of `priority_v0`.
something
  return stuff

but somehow, when evaluating, the code grabs the priority_v0 from the output instead of the priority_v1.
I suspect this is due to the fact that I use a different LLM. What should I look for in evaluator.py to adapt the code?

Well, I eventually found out the way.
I had to modify this function and this function, as well as understand the type of output my LMM was producing.

For those with LLM that have different input, see also this comment

BTW: This problem was solved in the commit


Repeatability

This addresses the issue for better seeding  #253 .
The trick was two create two graphs since in a single graph two random variable nodes must generate different values.
There are still some repeatability issues but only GPU and of a lesser magnitude. CPU appears totally deterministic now.

Thanks David, LGTM

Did you get a chance to test the atol values on both GPU and CPU thoroughly after we spoke?

Thank you!


Added Visualizer class

Added Visualizer class to handle plots and other visualizations.  Added
code to call it from maneuver

Thanks gurgentus, I'll merge it when I get to my laptop.
Once we have tests passing we will move this code to the Udacity repo.


store bug fix: ReplicationJobRunner should always mark running as false after the function exits

runner.running was not changed in some error return cases. This patches fixes the bug. Also added some new metrics.

Coverage decreased (-0.2%) to 68.107% when pulling d305b8ac19cff3f9b64d0d00c9664ab40a8cd1d0 on r_f into c12bb7fbe1a7bd806f87cd0ca7451cd1dcaa7ad3 on master.


🔧 Loaders and util for H5 and NIfTI transforms

Started at the NiPreps hackathon at MIT in May 2024.
Thanks again to @jmarabotto and @oesteban + @effigies for catching me up to speed and helping orient me on the nitransforms latest.
✅ This PR:

Adds loaders for .h5 and NIfTI (.nii or .nii.gz) transform files.
Adds a function for .x5 reading which only calls the H5 loader for now; set to be expanded as needed.
Adds an xfm_util level to the CLI which will read in a transform file and internally represent it as an instance of the TransformBase class, forming the X5 structure.

For now, this simply prints out some information, which can be used to verify the transform was read in properly.

Moves all I/O and I/O-related package/API calls to nitransforms/io/base.py

Starts the process of keeping the internal logic of the X5 transform structures and functions clean so that it can be unit tested without any actual transform files.

⚠ Needs attention:

Functions exist to write out the X5 structure/transform information to either HDF5 and X5 files, but these need to be updated and completed.
More cleanup of dependencies on the lower-level details from the higher-level logic.

❓ Questions:

I don't think the internal structure of the X5 hierarchy is complete or accurate. I started with @jmarabotto 's great flowchart as a guide, but I know this might have been in flux.
In keeping with the flat namespace of the HDF5/X5 structure, the key levels remain flat (i.e. TransformGroup/0/Transform but I'm not sure if some of the final levels, like /Transform, should be a new sub-level/sub-dictionary or if they should directly contain the NumPy array defining the actual transform, etc.

▶ Next steps (that I have in mind but I defer to the nitransforms contributors):

Completion of the write-out functions.
Some validation showing that converting any transform to an .x5 transform and applying it produces the exact same result as using the original (ex. ANTs, FSL etc.) transform with its appropriate application tool.

Will have a look ASAP!

@oesteban shall I merge into my fork and push this PR across to the main Nitransforms repo?

@jmarabotto sure, let's do that. Please propagate (and possibly update) @sgiavasis' initial message into the new PR

Hi @sgiavasis @oesteban , I have merged this and propagated to the main nitransforms repo. Lmk if I can help in any way! Thanks again for the work on this.


Support for Laravel 9

add support for laravel 9
fixes #49

Can this be tagged for release? Thanks!

how can i do the release ?

ping @pulkitjalan

@pulkitjalan can you please do the release for this merge?


support unknown and Record

Fixes https://github.com/Azure/autorest.csharp/issues/3243
Fixes https://github.com/Azure/autorest.csharp/issues/3244
Description
Per our discussed design,

unknown in typespec => BinaryData
Record<unknown> in typespec => IDictionary<string, BinaryData> when it is not readonly, IReadOnlyDictionary<string, BinaryData> when it is readonly.

Checklist
To ensure a quick review and merge, please ensure:

[ ] The PR has a understandable title and description explaining the why and what.
[ ] The PR is opened in draft if not ready for review yet.

If opened in draft, please allocate sufficient time (24 hours) after moving out of draft for review

[ ] The branch is recent enough to not have merge conflicts upon creation.

Ready to Land?

[ ] Build is completely green

Submissions with test failures require tracking issue and approval of a CODEOWNER

[ ] At least one +1 review by a CODEOWNER
[ ] All -1 reviews are confirmed resolved by the reviewer

Override/Marking reviews stale must be discussed with CODEOWNERS first

@m-nash could you have a review on this PR?

@ArcturusZhang, please help to regen and check the SDK change before merging this PR.

@ArcturusZhang, please help to regen and check the SDK change before merging this PR.

The regen only has some document changes in schema descriptions. We need to refine those anyway.


CLI: many commands missing compared to indy-node CLI

According to the indy-node readme, the indy-sdk cli should replace the indy-node cli.
But, looking at the source of the former, the source of the latter and the design document of the latter, it appears that the indy-sdk cli is still missing many functions.
I found a jira issue about the overall switch between the two clis, but I didn't manage to understand whether the missing commands will be implemented and, if so, whether they will be available soon or require some more time.

I noticed you were able to talk with @dhh1128 about this on RocketChat. Is there anything you can add here to clarify what seems to be missing?

According to the documentation in indy-node, the CLI implemented there has commands to perform the following actions, which are currently missing from the CLI in this repository:

load a JSON file containing a connection request (load <file>)
show a connection / connection request (show connection <name>)
accept a connection request (accept request from <name>)
check secure agent-to-agent communication channel (ping <name>)
show a claim from the wallet or available from a remote agent (show claim <name>)
obtain a claim from a remote agent (request claim <name>)
show a proof request (show proof request <name>)
send a proof to a remote agent (send proof <name> to <name>)

Most of these commands require the implementation of agent-to-agent communication, which is implemented in indy-node/indy_client (reusing some code from indy-plenum), but not in libindy.

We are using Jira to track issues now, please use this Jira IS-995 ticket for any further discussion.


Position: fixed sidebars issue

Hello there,
I'm not much of a developer, but I created this website in Webflow, which includes a section with a list of items. Each item opens up its own sidebar as you click on them. Initially, the sidebar is moved 100% to the right and then slides in once the item is clicked. However, when the sidebar moves into the screen, it contains content that overlaps 100vh and should be scrollable. Unfortunately, when I install Lenis and attempt to scroll through this sidebar, my body moves, but the sidebar remains static, and nothing happens. Removing Lenis from my code resolves the issue, and everything works as intended.
I really love this plugin, as it offers the best smooth scrolling experience available, so I'd greatly appreciate any assistance.
Here's the link to my live website, which might provide further insight: mediteranskisan.com

Well... a few minutes later, I found a solution...
Just added data-lenis-prevent to a parent which holds each individual sidebar


Merge devel into master

Description
Checklist/ToDo's

[ ] Added documentation?
[ ] Added release note entry? (  docs/release-notes.md )
[ ] Tested?

[ ] Azure DNS tests: work in progress

Azure DNS Testing
TASK [openshift-4-cluster : Create ClusteRole nfs-client-provisioner-runner] ******************************************************************************************
Monday 02 November 2020  13:46:29 +0100 (0:00:03.313)       0:19:51.171 *******
fatal: [localhost]: FAILED! => {"changed": false, "msg": "Failed to find exact match for rbac.authorization.k8s.io/v1.ClusterRole by [kind, name, singularName, shortNa
mes]"}

=> strange, looks like that is not related to azure dns. Looks like a strange RC. Rerun ./ansible/02-create-cluster.yml


Set tags for correctness test targets that run on external clusters.

This ensures that the tests run exclusively and without regard to previously cached test results.

This change is


Make wormhole server accept multiple tunnel protocols

Right now the wormhole server is configured to accept one tunnel protocol (e.g. TCP/TLS conn pool or SSH). Given that we're working on adding HTTP2 tunneling (#8), it would be nice if various clients could just connect to the server.
Client-side:

can connect to the server and pass config info
display error/warning info for unsupported config
could include auth to check if token is valid

Server-side:

validate client request
provide endpoint info for client to connect to (e.g. SSH server, HTTP2 server, TLS server)
could check for client versions to determine if they're still supported.

being addressed at https://github.com/doodles526/wormhole/tree/handler-rewrite


Added contribution instructions

resolves #86

@cjwilliams @cat-haines Can you please review this and make sure this all looks okay?

The language and information in this guide is really great, thanks for putting it together!!
One addition that might be nice to see is where developers can go for support while developing for Clay (#clay on Slack, @pebbledev, forums, etc)

@cat-haines okay updated :)

@cat-haines updated.

:+1: @keegan-lillo - One tiny comment on the "Getting Help" section, but this looks good.
You may also want to link to the contributing guide from the README.md to make it more discoverable.


chore(doc:formatters): fix typo

As found in https://github.com/Seldaek/monolog/pull/1504#discussion_r501172073.

Thanks


Expand on range types changesets supports

Currently changesets supports 1.0.0, ~1.0.0, and ^1.0.0, which is a subset of valid semver definitions.
@mitchellhamilton found the range types that yarn supports, and suggested we should do that.
So we need to: refind that info (I have lost the link he gave me soz)
Align our range type function with it.

https://yarnpkg.com/lang/en/docs/cli/upgrade/#toc-yarn-upgrade-package-latest-l-caret-tilde-exact-pattern


When manually setting the value of AnimationController, the status is always forward.

Use case
In the source code, when setting a new value, _direction is used to change _status. If _direction is not changed by other means, _direction defaults to _AnimationDirection.forward, resulting in status also being AnimationStatus.forward. In some cases, I need status to be AnimationStatus.reverse when manually setting value, which is not possible at this time.
https://github.com/flutter/flutter/blob/5a3fd44aec1ff9c8d57fd7ddfd27e97099a05c13/packages/flutter/lib/src/animation/animation_controller.dart#L431-L434
Proposal
Hope to provide similar methods:
void setValue(double value, [AnimationDirection? direction]);

Usage scenarios
When you manually slide back, reverseBuilder cannot be triggered, but forwardBuilder is called. The root cause is that status is forward when AnimationController.value = x

class FooRoute<T> extends PageRoute<T> {

  @override
  Widget buildTransitions(
    BuildContext context,
    Animation<double> animation,
    Animation<double> secondaryAnimation,
    Widget child,
  ) {
    return DualTransitionBuilder(
      animation: animation,
      forwardBuilder: (context, animation, child) {
        return FadeTransition(opacity: animation, child: child);
      },
      reverseBuilder: (context, animation, child) {
        return CupertinoRouteTransitionMixin.buildPageTransitions(
          this,
          context,
          ReverseAnimation(animation),
          secondaryAnimation,
          child!,
        );
      },
      child: child,
    );
  }
}

cc @nate-thegrate

I apologize @fingerart, I think I'll be closing https://github.com/flutter/flutter/pull/158542 since it causes regressions in some existing checks. If you'd like to pick it up and work through the breakages, feel free to do so and I'd be happy to review (same offer applies to anyone else reading this issue)!

Is there any workaround for now?


HDDS-8450. Dedicated acceptance test suite for s3a

What changes were proposed in this pull request?
Run some S3A contract tests as part of acceptance check.  Tests are executed via Maven against Docker Compose-based Ozone cluster, using Hadoop 3.3.6 source.  JUnit test results and output for failed tests are included in the artifact, similar to integration check.
Limitations:

only works in unsecure env. for now
tests that currently fail for various reasons are excluded (created Jira issues for each)

https://issues.apache.org/jira/browse/HDDS-8450
How was this patch tested?
CI:
https://github.com/adoroszlai/ozone/actions/runs/8484286662
Also tried some failing tests locally to verify behavior of the check.

@ivandika3 Could you please review, too?

@adoroszlai @errose28  I have a question regarding S3A in general. In what case does using S3A protocol more beneficial rather than using the OFS protocol? My understanding is that S3A aims to provide HDFS semantic to non hadoop-compatible storage (like using S3Guard when AWS S3 was still eventually consistent), whereas Ozone already provides OFS as a direct implementation of HDFS filesystem.
From the performance point of view, it seems using s3a adds more network overhead since the s3a client needs to connect to S3G, and the S3G connect to OM whereas OFS will directly connect to OM.
Thanks in advance.

In what case does using S3A protocol more beneficial rather than using the OFS protocol?

Another case may be if client already has jars required for S3A, but classpath cannot be changed to add Ozone FS jar.  Not sure if that's a common situation.

Thanks @errose28 for the review.  Patch is updated, please take another look.

Thanks @errose28, @ivandika3 for the review.


apps-sc: Fixed storage leak in log-manager compaction

[!warning]
This is public repository, ensure not to disclose:

[ ] personal data beyond what is necessary for interacting with this pull request
[ ] business confidential information, such as customer names

What kind of PR is this?
Required: Mark one of the following that is applicable:

[ ] kind/feature
[ ] kind/improvement
[ ] kind/deprecation
[ ] kind/documentation
[ ] kind/clean-up
[x] kind/bug
[ ] kind/other

Optional: Mark one or more of the following that are applicable:

[!important]
Breaking changes should be marked kind/admin-change or kind/dev-change depending on type
Critical security fixes should be marked with kind/security

[ ] kind/admin-change
[ ] kind/dev-change
[ ] kind/security
[ ] kind/adr

Platform Administrator notice
Fixes issue where log-manager compaction script isn't removing all files as it should

What does this PR do / why do we need this PR?

...

Fixes #

Additional information to reviewers
Apparently the --rm flag for zstd doesn't seem to be reliable (don't ask me why). So I just replaced them with rm commands instead afterwards. And also removed the index folder at the end just to be sure
Screenshots
Checklist

[x] Proper commit message prefix on all commits

Change checks:

[ ] The change is transparent
[ ] The change is disruptive
[ ] The change requires no migration steps
[ ] The change requires migration steps

Metrics checks:

[ ] The metrics are still exposed and present in Grafana after the change
[ ] The metrics names didn't change (Grafana dashboards and Prometheus alerts are not affected)
[ ] The metrics names did change (Grafana dashboards and Prometheus alerts were fixed)

Logs checks:

[ ] The logs do not show any errors after the change

Network Policy checks:

[ ] Any changed pod is covered by Network Policies
[ ] The change does not cause any dropped packages in the NetworkPolicy Dashboard

Pod Security Policy checks:

[ ] Any changed pod is covered by Pod Security Admission
[ ] Any changed pod is covered by Gatekeeper Pod Security Policies
[ ] The change does not cause any pods to be blocked by Pod Security Admission or Policies

Falco checks:

[ ] The change does not cause any alerts to be generated by Falco

Audit checks:

[ ] The change does not cause any unnecessary Kubernetes audit events
[ ] The change requires changes to Kubernetes audit policy

Bug checks:

[ ] The bug fix is covered by regression tests

Hmm, maybe we need a deeper investigation here. Because it seems like the files are still there even when using this script :thinking:  Could it be something with the filesystem of the volume? Maybe I can try to add a sync somewhere to force it to sync the fs

Any updates here :) ?

Tested and it seems to work wonders. From 150GB+ to 5-6GB. But would like another approve


docs: Add OpenDigger badge.

Signed-off-by: frank-zsy<EMAIL_ADDRESS>Brief Information
This pull request is in the type of (more info about types):

[ ] build
[ ] ci
[x] docs
[ ] feat
[ ] fix
[ ] perf
[ ] refactor
[ ] test

Details

Add OpenDigger project badge and link as OpenDigger suggest.
Checklist

[x] I have read the CONTRIBUTING doc
[x] I have signed the CLA

Others

Wow, the badge looks nice! Yet we also need to put it into Chinese README~

Right, I forgot this. Sorry.


Add favicon

These changes are identical to how favicons work for stripe.com.
r? @atty-stripe

LGTM

LGTM


[BUG] DriverAhead not working properly

Describe the issue:
I noticed when Norris was leading behind safetycar Verstappen "overtook" several times Norris.
Reproduce the code example:
Date	                            DriverAhead	DistanceToDriverAhead	Time Distance	RelativeDistance Status Driver
2024-03-09 17:20:17.2220000	0	1.42639	0 days 00:01:41.705000	3905.45	0.635869	OnTrack	NOR
2024-03-09 17:20:17.3820000	0	1.42639	0 days 00:01:41.865000	3913.27	0.637142	OnTrack	NOR
2024-03-09 17:20:17.3910000	0	1.42639	0 days 00:01:41.874000	3913.7	0.637213	OnTrack	NOR
2024-03-09 17:20:17.5420000	0	1.42639	0 days 00:01:42.025000	3920.91	0.638387	OnTrack	NOR
2024-03-09 17:20:17.6510000	0	1.42639	0 days 00:01:42.134000	3926.07	0.639227	OnTrack	NOR
2024-03-09 17:20:17.7020000	0	1.42639	0 days 00:01:42.185000	3928.47	0.639617	OnTrack	NOR
2024-03-09 17:20:17.9510000	0	1.42639	0 days 00:01:42.434000	3940	0.641494	OnTrack	NOR
2024-03-09 17:20:18.0620000	0	1.42639	0 days 00:01:42.545000	3945.07	0.64232	OnTrack	NOR
2024-03-09 17:20:18.1710000	0	1.42639	0 days 00:01:42.654000	3950.02	0.643126	OnTrack	NOR
2024-03-09 17:20:18.3310000	0	1.42639	0 days 00:01:42.814000	3957.22	0.644299	OnTrack	NOR
2024-03-09 17:20:18.3310000	4	27.6003	0 days 00:01:41.715000	3929.31	0.638692	OnTrack	VER
2024-03-09 17:20:18.4220000	0	1.42639	0 days 00:01:42.905000	3961.27	0.644957	OnTrack	NOR
2024-03-09 17:20:18.4220000	4	27.0003	0 days 00:01:41.806000	3934	0.639454	OnTrack	VER
2024-03-09 17:20:19.1710000	0	1.42639	0 days 00:01:43.654000	3993.33	0.650177	OnTrack	NOR
2024-03-09 17:20:19.1710000	4	21.6603	0 days 00:01:42.555000	3975.95	0.646274	OnTrack	VER
2024-03-09 17:20:19.2230000	0	1.42639	0 days 00:01:43.706000	3995.53	0.650536	OnTrack	NOR
2024-03-09 17:20:19.2230000	4	16.3203	0 days 00:01:42.607000	3978.94	0.64676	OnTrack	VER
2024-03-09 17:20:19.3510000	0	1.42639	0 days 00:01:43.834000	4000.96	0.65142	OnTrack	NOR
2024-03-09 17:20:19.3510000	4	14.4869	0 days 00:01:42.735000	3986.3	0.647956	OnTrack	VER
2024-03-09 17:20:19.4630000	0	1.42639	0 days 00:01:43.946000	4005.73	0.652197	OnTrack	NOR
2024-03-09 17:20:19.4630000	4	12.6536	0 days 00:01:42.847000	3992.81	0.649014	OnTrack	VER
2024-03-09 17:20:19.5510000	0	1.42639	0 days 00:01:44.034000	4009.5	0.65281	OnTrack	NOR
2024-03-09 17:20:19.5510000	4	10.6536	0 days 00:01:42.935000	3998.01	0.649859	OnTrack	VER
2024-03-09 17:20:19.7030000	0	1.42639	0 days 00:01:44.186000	4016.07	0.653879	OnTrack	NOR
2024-03-09 17:20:19.7030000	4	8.65361	0 days 00:01:43.087000	4007.14	0.651344	OnTrack	VER
2024-03-09 17:20:19.8310000	0	1.42639	0 days 00:01:44.314000	4021.69	0.654794	OnTrack	NOR
2024-03-09 17:20:19.8310000	4	6.93139	0 days 00:01:43.215000	4014.97	0.652616	OnTrack	VER
2024-03-09 17:20:19.9030000	0	1.42639	0 days 00:01:44.386000	4024.9	0.655318	OnTrack	NOR
2024-03-09 17:20:19.9030000	4	5.20917	0 days 00:01:43.287000	4019.42	0.65334	OnTrack	VER
2024-03-09 17:20:20.0710000	0	1.42639	0 days 00:01:44.554000	4032.59	0.65657	OnTrack	NOR
2024-03-09 17:20:20.0710000	4	3.30917	0 days 00:01:43.455000	4029.88	0.65504	OnTrack	VER
2024-03-09 17:20:20.1430000	0	1.42639	0 days 00:01:44.626000	4035.97	0.65712	OnTrack	NOR
2024-03-09 17:20:20.1430000	4	1.40917	0 days 00:01:43.527000	4034.29	0.655756	OnTrack	VER
2024-03-09 17:20:20.2110000	0	1.42639	0 days 00:01:44.694000	4039.19	0.657645	OnTrack	NOR
2024-03-09 17:20:20.2110000	4	1.33361	0 days 00:01:43.595000	4038.37	0.656419	OnTrack	VER
2024-03-09 17:20:20.3910000	0	1.42639	0 days 00:01:44.874000	4047.91	0.659063	OnTrack	NOR
2024-03-09 17:20:20.3910000	4	1.25806	0 days 00:01:43.775000	4048.72	0.658102	OnTrack	VER
2024-03-09 17:20:20.4620000	1	1.42639	0 days 00:01:44.945000	4051.39	0.65963	OnTrack	NOR
2024-03-09 17:20:20.4620000	0	1.1825	0 days 00:01:43.846000	4052.54	0.658723	OnTrack	VER
2024-03-09 17:20:20.6110000	1	1.35972	0 days 00:01:45.094000	4058.76	0.66083	OnTrack	NOR
2024-03-09 17:20:20.6110000	0	1.06917	0 days 00:01:43.995000	4060.03	0.65994	OnTrack	VER
2024-03-09 17:20:20.7020000	1	1.29306	0 days 00:01:45.185000	4063.32	0.661573	OnTrack	NOR
2024-03-09 17:20:20.7020000	0	0.955833	0 days 00:01:44.086000	4064.34	0.660641	OnTrack	VER
2024-03-09 17:20:20.8110000	1	1.31337	0 days 00:01:45.294000	4068.86	0.662475	OnTrack	NOR
2024-03-09 17:20:20.8110000	0	0.8425	0 days 00:01:44.195000	4069.33	0.661452	OnTrack	VER
2024-03-09 17:20:20.9820000	0	1.33369	0 days 00:01:45.465000	4077.79	0.663928	OnTrack	NOR
2024-03-09 17:20:20.9820000	4	0.729167	0 days 00:01:44.366000	4076.79	0.662664	OnTrack	VER
2024-03-09 17:20:21.0910000	0	1.35401	0 days 00:01:45.574000	4083.69	0.664889	OnTrack	NOR
2024-03-09 17:20:21.0910000	4	3.47917	0 days 00:01:44.475000	4081.34	0.663405	OnTrack	VER
2024-03-09 17:20:21.3420000	0	1.37433	0 days 00:01:45.825000	4097.79	0.667185	OnTrack	NOR

Error message:
No response

This may be due to another bug, which causes the lap alignment to be skipped for the Saudi Arabian race (see #549)

@pesaventofilippo that's a good assumption, especially if this problem is not present in other races. I haven't investigated the problem here yet, though.
In general, given the very short distances in the safety car train and the fairly low resolution source data, this may always be tricky.
In case someone wants to try and fix #549, I'm happy to give some guidance. But the math behind that can make your head spin quite a bit 😅

@pesaventofilippo @theOehrly not sure but I checked Bahrain Grand Prix and got a bit of the issues. As example; Data is from lap 4

Uhm, I tried testing for HAM and PIA on lap 4, here's what I found:
import fastf1

session = fastf1.get_session(2024, "Bahrain", "R")
session.load()

ham_lap = session.laps.pick_driver("HAM").pick_lap(4)
ham_car = ham_lap.get_car_data().add_driver_ahead()

pia_lap = session.laps.pick_driver("PIA").pick_lap(4)
pia_car = pia_lap.get_car_data().add_driver_ahead()

# Print all instances in which the DriverAhead for HAM isn't PIA (as it should be)
print(ham_car[ham_car.DriverAhead != "81"])

# Print all instances in which the DriverAhead for PIA isn't ALO (as it should be)
print(pia_car[pia_car.DriverAhead != "14"])

Output:
                     Date    RPM  ...  DriverAhead  DistanceToDriverAhead
0 2024-03-02 15:08:42.470  10410  ...                                 NaN

                    Date    RPM  ...  DriverAhead  DistanceToDriverAhead
0 2024-03-02 15:08:41.910  10448  ...                                 NaN

In both cases, the only instance in which the DriverAhead isn't correct is the first row in the lap data, where DriverAhead is 0 (which is to be expected for the first row).
Maybe there's a problem in the way you determine the position during a lap? I could have missed something or maybe I'm not understanding, correct me if I'm wrong!

I used Distance and RelativeDistance since DriverAhead is not giving proper info. As example Bahrain;
can it be the source of data (car, pos, interpolation)???


Fixed Typo in DevGuide.md

Fixed a heading for a command, windows command was written under macOS.

Duplicate of #345

Closed as duplicate.


Error message when teams and teams roles csv files are empty

When the teams and teams roles csv files are empty the following error messages is show.
To reproduce remove all team names and team roles from their respective csv files.
 Error: Insufficient roles blocks │ │   on resources.tf line 60, in resource "anypoint_team_roles" "lvl1_teams_roles": │   60: resource "anypoint_team_roles" "lvl1_teams_roles" { │ │ At least 1 "roles" blocks are required.

this issue is related to the template and therefore should not be opened here. closing this one


Deploy files to Remote on Source or F5

Also sync command on the project if user doesn't use F5.

Remote F5 Completed. Remote source pending.


Duplicate 'general' channel in team

I have a team where I accidentally clicked to make it a "big team" (was a long time ago, so not sure exactly what that process was).
Now I have 2 #general channels, and both are treated as the main channel that cannot be deleted or renamed. I also occasionally see errors that complain about the channel being in use or something like that.

I'd prefer this to be a "small team" i.e. only one channel.
Anything I can do here to remove one of the channels (either one, doesn't matter)?

Fixed in https://github.com/keybase/client/pull/20617 and resolved over chat.


Transaction Data Gathered by the fabric-ccp Adapter

Expected Behavior
When I'm running the fabric / fabric-ccp adapter, I expect to receive some transaction specific custom data per transaction, as described in the docs.
https://hyperledger.github.io/caliper/docs/Fabric_Ccp_Configuration.html
Actual Behavior
There's no custom data about the transaction, when tested with Fabric v1.2 to v1.4. Enabling such custom data about transactions on the network is not described in the documentation.
Possible Fix
Pointing to a documentation / setting in the Hyperledger Fabric docs for enabling said behaviour, or perhaps adding more detail on the corresponding section of Caliper documentation.
@aklenik

@ig0rsky As detailed by the CCP docs, the return value of the invoke* functions is an array of TxStatus instances, that provide some functions to retrieve Tx data. You call the invoke* functions from your user callback modules, so you can inspect the gathered data there. Automatic export of gathered data is not provided yet.
Something like this should work in your user callback module please verify it:
let results = bc.invokeSmartContract(...);
for (let result of results) {
  console.log(result.GetID());
  console.log(result.GetStatus());
  console.log(result.GetTimeCreate());
  console.log(result.GetTimeFinal());

  for (let customEntry of result.GetCustomData()) {
    console.log(`${customEntry[0]}: ${customEntry[1]}`);
  }
}

Thank you, I was turning the object directly into JSON using JSON.stringify and wondering why some of the data is not there. If I'm not mistaken, the old(fabric) query API still won't return any custom data, correct? @aklenik

@ig0rsky IIRC only the basics. We plan to deprecate that adapter, so it's better to stick with the Fabric-CCP adapter

Sorry sir, i can not find Fabric cpp adapter. Would you give me some suggest? Thank you

@hoannguyen26196 The CCP adapter has been renamed simple to the Fabric adapter (and the old one was removed). You can refer to this doc section for the gathered data: https://hyperledger.github.io/caliper/vLatest/fabric-config/#gathered-tx-data


Add API for alert cleanup

What is the feature request? What problem does it solve?
The Control Service sends notification emails for completed data job executions. The notifications are based on alerts that are themselves based on metrics exposed by the Control Service for individual data jobs. If a job is disabled, the respective alert remains active forever and this may cause noise in monitoring.
As a Control Service operator, I would like to have an option to reset such alerts manually by using the Control Service's API.
Suggested solution
Provide an API to reset alerts for specific jobs.
Acceptance criteria
I am able to cancel active alerts by using the API of the Control Service.

Triaged. Closed as stale.


No information is given on stdin for pre/post-receive

Gitea version (or commit ref): 1.1.1
Git version: 2.13.0.windows.1
Operating system: Windows Server 2012 R2
Database (use [x]):

[ ] PostgreSQL
[x] MySQL
[ ] MSSQL
[ ] SQLite

Can you reproduce the bug at https://try.gitea.io:

[ ] Yes (provide example URL)
[ ] No
[x] Not relevant

Log gist:

Description
pre-receive and post-receive git hooks are supposed to receive information on stdin
related #1808

I've noted the same issue when I migrated from my 0.9 Gogs install to Gitea.
This comes from the new post-receive wrapper script in hooks/post-receive
#!/usr/bin/env bash
ORI_DIR=`pwd`
SHELL_FOLDER=$(cd "$(dirname "$0")";pwd)
cd "$ORI_DIR"
for i in `ls "$SHELL_FOLDER/post-receive.d"`; do
    sh "$SHELL_FOLDER/post-receive.d/$i"
done

Your actual hook is in hooks/post-receive.d/post-receive
This indeed doesn't forward stdin to post-receive.d/ scripts.
Something like that would make your pre/post-receive work again:
#!/usr/bin/env bash
ORI_DIR=`pwd`
SHELL_FOLDER=$(cd "$(dirname "$0")";pwd)
read STDIN
cd "$ORI_DIR"
for i in `ls "$SHELL_FOLDER/post-receive.d"`; do
    echo "$STDIN" | sh "$SHELL_FOLDER/post-receive.d/$i"
done

I'll see if I can make a PR for that.

We need also some integration tests about hooks.

This issue is fixed in master with #1257 and duplicate of #1250

Also #1257 was backported to 1.1.x version but not migration. You should manually run Resync pre-receive, update and post-receive hooks of all repositories. from administrator dashboard to have correct hooks


Android Basics: Fragments Navigation component

URL of codelab
https://developer.android.com/codelabs/basic-android-kotlin-training-fragments-navigation-component?continue=https%3A%2F%2Fdeveloper.android.com%2Fcourses%2Fpathways%2Fandroid-basics-kotlin-unit-3-pathway-2%23codelab-https%3A%2F%2Fdeveloper.android.com%2Fcodelabs%2Fbasic-android-kotlin-training-fragments-navigation-component#4
In which task and step of the codelab can this issue be found?
Task 2
Step 5
Describe the problem
The code provided by the codelab for onCreateView (), throws an error in my Android Studio
Steps to reproduce?

Go to... th URL
Click on... Task 2 and later Step 5
See error...The code shown for point 4 of the creation of the LetterListFragment, specifically for override onCreateView () throws an error in Android Studio

Versions
Android Studio version: 4.1
API version of the emulator: Physical Device
Additional information
Include screenshots if they would be useful in clarifying the problem.

oppp!! Sorry my bad, I had misdeclared the type of _binding

@nokyslash
Thank you for reaching out to us.
Let us know if you need further assistance.


HBASE-24096 Fix TestRSGroupsBalance

if hbase.rsgroup.table.mapping.script not set, we should not return DEFAULT_GROUP...

May I ask what will happen when returning "default" in TestRSGroupsBalance

If we return "default", we will not fall back to namespace as rsgroup if exists.  @Reidddddd
" // Finally, we will try to fall back to namespace as rsgroup if exists "

Oh, get it.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 31s
Docker mode activated.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
6m 49s
the patch passed

+1 :green_heart:
compile
1m 17s
the patch passed

+1 :green_heart:
javac
1m 17s
the patch passed

+1 :green_heart:
shadedjars
6m 28s
patch has no errors when building our shaded downstream artifacts.

-0 :warning:
javadoc
0m 46s
hbase-server in the patch failed.

_ Other Tests _

+1 :green_heart:
unit
68m 38s
hbase-server in the patch passed.

102m 12s

Subsystem
Report/Notes

Docker
Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1402/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/1402

Optional Tests
javac javadoc unit shadedjars compile

uname
Linux 5844934dd2b9 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / 6f213e9d5a

Default Java
2020-01-14

javadoc
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1402/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt

javadoc
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1402/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt

Test Results
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1402/1/testReport/

Max. process+thread count
5779 (vs. ulimit of 10000)

modules
C: hbase-server U: hbase-server

Console output
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1402/1/console

versions
git=2.17.1 maven=2018-06-17T18:33:14Z)

Powered by
Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.


Vision API Dependency Issue

Hi,
I am trying to integrate Google Vision API into a UI on our IIS-based system, but I have run into an issue getting the API calls to work. To be clear, I have the API working successfully on a console application. However, in a separate use case, we need to make a .dll file for a UI. For this use case, we keep getting a dependency conflict for Google.Apis.Auth, expecting <IP_ADDRESS>, but we have <IP_ADDRESS>. The API was installed using Nuget, so I am unsure why this conflict is happening. We also had to add the assembles to the GAC to resolve a different issue. The solutions I have read in threads such as this have not worked.

We also had to add the assembles to the GAC to resolve a different issue.

That sounds like it may well be part of the problem. That's certainly a scenario we haven't tested. I would strongly advise removing the Google assemblies from the GAC. We can have a look at this "different issue" separately.
Please then describe your situation in more detail, ideally with a way we can reproduce the problem. Note that the application should pick up all the dependencies via NuGet, or be configured with appropriate assembly binding redirects. If you've used NuGet to create a class library, but then added a direct file reference to the DLL from your UI project, that could explain the problem.

When removing the assemblies from the GAC, we face a different but similar issue. Instead of there being a FileNotFoundException due to a versioning conflict with Google.Apis.Auth, a FileNotFoundException is thrown for Google.Cloud.Vision.V1 itself. The reason why we believed adding the assemblies to the GAC to be the solution is that when we add Google.Cloud.Vision.V1 to the GAC, a different assembly has this error. We go down the tree and arrive at having them all in the GAC. Once they're all in, this is when we encounter the error this thread was posted about.
For reference, the call we are testing this error with is simply
`ImageAnnotatorClient client = ImageAnnotatorClient.Create();
The exact error is:
System.IO.FileNotFoundException was unhandled by user code
HResult=-2147024894
Message=Could not load file or assembly 'Google.Cloud.Vision.V1, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=185c282632e132a0' or one of its dependencies. The system cannot find the file specified.
Source=GoogleVisionLinks
FileName=Google.Cloud.Vision.V1, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=185c282632e132a0
FusionLog==== Pre-bind state information ===
LOG: DisplayName = Google.Cloud.Vision.V1, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=185c282632e132a0
(Fully-specified)
LOG: Appbase = file:///E:/Adam/Studios/Products/
LOG: Initial PrivatePath = E:\Adam\Studios\Products\bin
Calling assembly : GoogleVisionLinks, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=null.
LOG: This bind starts in LoadFrom load context.
WRN: Native image will not be probed in LoadFrom context. Native image will only be probed in default load context, like with Assembly.Load().
LOG: Using application configuration file: E:\Adam\Studios\Products\web.config
LOG: Using host configuration file: C:\Windows\Microsoft.NET\Framework64\v4.0.30319\aspnet.config
LOG: Using machine configuration file from C:\Windows\Microsoft.NET\Framework64\v4.0.30319\config\machine.config.
LOG: Post-policy reference: Google.Cloud.Vision.V1, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=185c282632e132a0
LOG: Attempting download of new URL file:///C:/Windows/Microsoft.NET/Framework64/v4.0.30319/Temporary ASP.NET Files/products/ad62903d/e2049733/Google.Cloud.Vision.V1.DLL.
LOG: Attempting download of new URL file:///C:/Windows/Microsoft.NET/Framework64/v4.0.30319/Temporary ASP.NET Files/products/ad62903d/e2049733/Google.Cloud.Vision.V1/Google.Cloud.Vision.V1.DLL.
LOG: Attempting download of new URL file:///E:/Adam/Studios/Products/bin/Google.Cloud.Vision.V1.DLL.
LOG: Attempting download of new URL file:///E:/Adam/Studios/Products/bin/Google.Cloud.Vision.V1/Google.Cloud.Vision.V1.DLL.
LOG: Attempting download of new URL file:///C:/Windows/Microsoft.NET/Framework64/v4.0.30319/Temporary ASP.NET Files/products/ad62903d/e2049733/Google.Cloud.Vision.V1.EXE.
LOG: Attempting download of new URL file:///C:/Windows/Microsoft.NET/Framework64/v4.0.30319/Temporary ASP.NET Files/products/ad62903d/e2049733/Google.Cloud.Vision.V1/Google.Cloud.Vision.V1.EXE.
LOG: Attempting download of new URL file:///E:/Adam/Studios/Products/bin/Google.Cloud.Vision.V1.EXE.
LOG: Attempting download of new URL file:///E:/Adam/Studios/Products/bin/Google.Cloud.Vision.V1/Google.Cloud.Vision.V1.EXE.
LOG: Attempting download of new URL file:///E:/Adam/WorkingFolder/Assembly Cache/Google.Cloud.Vision.V1.DLL.
LOG: Attempting download of new URL file:///E:/Adam/WorkingFolder/Assembly Cache/Google.Cloud.Vision.V1/Google.Cloud.Vision.V1.DLL.
LOG: Attempting download of new URL file:///E:/Adam/WorkingFolder/Assembly Cache/Google.Cloud.Vision.V1.EXE.
LOG: Attempting download of new URL file:///E:/Adam/WorkingFolder/Assembly Cache/Google.Cloud.Vision.V1/Google.Cloud.Vision.V1.EXE.
StackTrace:
at GoogleVisionLinks.VisionLinkUI.CreateTable(Aspect a)
at GoogleVisionLinks.VisionLinkUI.CreateSubmitDialog(Aspect a)
at GoogleVisionLinks.VisionLinkUI.CreateChildControls()
at System.Web.UI.Control.EnsureChildControls()
at System.Web.UI.Control.PreRenderRecursiveInternal()
at System.Web.UI.Control.PreRenderRecursiveInternal()
at System.Web.UI.Control.PreRenderRecursiveInternal()
at System.Web.UI.Control.PreRenderRecursiveInternal()
at System.Web.UI.Control.PreRenderRecursiveInternal()
at System.Web.UI.Control.PreRenderRecursiveInternal()
at System.Web.UI.Control.PreRenderRecursiveInternal()
at System.Web.UI.Control.PreRenderRecursiveInternal()
at System.Web.UI.Control.PreRenderRecursiveInternal()
at System.Web.UI.Control.PreRenderRecursiveInternal()
at System.Web.UI.Control.PreRenderRecursiveInternal()
at System.Web.UI.Control.PreRenderRecursiveInternal()
at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)
InnerException:
Somehow this error causes such an issue that when debugging the method that the API call is in doesn't even get stepped into it. The API call is dozens of lines after the method definition, and the method doesn't seem to run at all when stepping line by line.`

Okay, let's stick with "not in GAC" mode (as that's the supported one). Please could you provide a way of me reproducing the problem? The exact version of the framework you're targeting, project type and dependency list would be really useful. Once I can reproduce the problem, it'll be much easier to help you.
You should also have a look at the generated .config file to see what assembly binding redirects are present.

You should also have a look at the generated .config file to see what assembly binding redirects are present.

Do you mean the app.config? The redirects present there look appropriate. As for the framework - we are on 4.5, the project is a class library, and we have used Nuget to install Vision API and its dependencies.

I decided to take an alternative route and have the API up and running without using a class library. I apologize for wasting your time.

@aschlumpf : What did you do to resolve this issue?