SFB Connector is going offline

Hi Team,
We have downloaded the connector and have hosted this scheduler which always keeps the user online on Azure web job. However, we are observing the user is going offline often and only after restart of web job, user is coming online. After enabling logs, below are the details that are captured and are not helping to dig in to the issue.

Would be helpful if you can provide any insight on this.

Hi @RajaniJillaGit
This error doesn't provide a lot of information. Any chance you could provide a bigger stack trace?

@baywet  @kenakamu
Getting the Same Issue , and this is the Stack Trace
System.NullReferenceException: Object reference not set to an instance of an object.     at Microsoft.Skype.UCWA.UCWAClient.d__421.MoveNext() in D:\UnileverUnabot\SFB\MultiPoolSFBConnector-Final\SFBConnector\UCWASDK\UCWAClient.cs:line 646
at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)     at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)     at System.Threading.Tasks.Task.Wait()     at SFBConnector.CustomConnector.Signin(String username, String password) in D:\UnileverUnabot\SFB\MultiPoolSFBConnector-Final\SFBConnector\TestClient\CustomConnector.cs:line 394


The documentation of the order of static constructors is wrong.

Type of issue
Other (describe below)
Description
The following bit seems wrong.

Base static constructors run. Any static constructors, starting with Object.Object through each base class to the direct base class.
The static constructor runs. The static constructor for the type runs.

As I will demonstrate, in cases where the members of the base class is not accessed by the deriving class, the deriving class constructor will run before the base constructor.
using static System.Console;

class BaseClass {
    static protected int c = 1;
    static BaseClass() => WriteLine($"{c++}. Static   constructor of *BaseClass*    called.");
    public BaseClass() => WriteLine($"{c++}. Instance constructor of *BaseClass*    called.");
}
class DerivedClass : BaseClass {
    static DerivedClass()          => WriteLine($"{c++}. Static   constructor of  DerivedClass  called.");
    public DerivedClass() : base() => WriteLine($"{c++}. Instance constructor of  DerivedClass  called.");
}

public static class C {
    public static int c = 1;
}
class BaseClass2 {
    static BaseClass2() => WriteLine($"{C.c++}. Static   constructor of *BaseClass2*   called.");
    public BaseClass2() => WriteLine($"{C.c++}. Instance constructor of *BaseClass2*   called.");
}
class DerivedClass2 : BaseClass2 {
    static DerivedClass2()          => WriteLine($"{C.c++}. Static   constructor of  DerivedClass2 called.");
    public DerivedClass2() : base() => WriteLine($"{C.c++}. Instance constructor of  DerivedClass2 called.");
}

public class Program {
    public static void Main() {
        var instance = new DerivedClass();
        WriteLine();
        var instance2 = new DerivedClass2();
    }
}

The results are different in the two cases:
1. Static   constructor of *BaseClass*    called.
2. Static   constructor of  DerivedClass  called.
3. Instance constructor of *BaseClass*    called.
4. Instance constructor of  DerivedClass  called.

1. Static   constructor of  DerivedClass2 called.
2. Static   constructor of *BaseClass2*   called.
3. Instance constructor of *BaseClass2*   called.
4. Instance constructor of  DerivedClass2 called.

Page URL
https://learn.microsoft.com/en-us/dotnet/csharp/programming-guide/classes-and-structs/static-constructors
Content source URL
https://github.com/dotnet/docs/blob/main/docs/csharp/programming-guide/classes-and-structs/static-constructors.md
Document Version Independent Id
ac4f6361-70a7-fb43-101e-c12044f60896
Article author
@BillWagner
Metadata

ID: aa3ed1c5-1f79-9234-beec-b5771c3fb3c0
Service: dotnet-csharp
Sub-service: fundamentals

Here's a related interesting bit:
using static System.Console;

class BaseClass {
    static protected int c = 1;
    static BaseClass() => WriteLine($"{c++}. Static   constructor of *BaseClass*    called.");
    public BaseClass() => WriteLine($"{c++}. Instance constructor of *BaseClass*    called.");
}
class DerivedClass : BaseClass {
    static DerivedClass()          => WriteLine($"{c++}. Static   constructor of  DerivedClass  called.");
    public DerivedClass() : base() => WriteLine($"{c++}. Instance constructor of  DerivedClass  called.");
}

public static class C {
    public static int c = 1;
}
class BaseClass2 {
    static BaseClass2() => WriteLine($"{C.c++}. Static   constructor of *BaseClass2*   called.");
    public BaseClass2() => WriteLine($"{C.c++}. Instance constructor of *BaseClass2*   called.");
}
class DerivedClass2 : BaseClass {
    static DerivedClass2()          => WriteLine($"{C.c++}. Static   constructor of  DerivedClass2 called.");
    public DerivedClass2() : base() => WriteLine($"{C.c++}. Instance constructor of  DerivedClass2 called.");
}

public class Program {
    public static void Main() {
        var instance = new DerivedClass();
        WriteLine();
        var instance2 = new DerivedClass2();
    }
}

The output is:
1. Static   constructor of *BaseClass*    called.
2. Static   constructor of  DerivedClass  called.
3. Instance constructor of *BaseClass*    called.
4. Instance constructor of  DerivedClass  called.

1. Static   constructor of  DerivedClass2 called.
5. Instance constructor of *BaseClass*    called.
2. Instance constructor of  DerivedClass2 called.

The static ctor for BaseClass isn't required to be run at all. No static members have been accessed.

Here's a related interesting bit:
...
The static ctor for BaseClass isn't required to be run at all. No static members have been accessed.

I think you just mixed up BaseClass and BaseClass2 there.


Unable to start training due to the gpu memory

Hi, thanks for the amazing job.
I'm trying to train a mq-groundingdino-t model with the object365 data. But seems like I don't have enough gpu memory. Would you mind telling me how much gpu memory I need to reserve for training of 1 batch?
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 80.00 MiB (GPU 0; 10.75 GiB                                                         total capacity; 9.34 GiB already allocated; 27.62 MiB free; 9.77 GiB reserved in total by PyTor                                                        ch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentat                                                        ion.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

Hello! Thank you for your interest in our work.
I have tried training MQ-Grounding-DINO using a batch size of 1, and it requires approximately 9.6GiB of RAM per GPU.

Hello! Thank you for your interest in our work.
I have tried training MQ-Grounding-DINO using a batch size of 1, and it requires approximately 9.6GiB of RAM per GPU.

In my case, it required more than 11g of gpu memory and show up that OutOfMemoryError. I already set the batch size as 1, and use the tiny backbone of the g-dino. And also the optimizer I change to sgd to make the training using less gpu memory. But still it doesn't work.
Is there any way I can reduce the usage of gpu memory?

Oh, maybe you can try the followings:

Set DATASETS.RANDOM_SAMPLE_NEG to a lower value (default 85), for example, 60. This controls the number of textual categories fed into the model in one forward pass, a low value results in a shorter input sequence length.
Maybe you can try gradient checkpointing. But I didn't implement this in the code :(

By the way, ensure that SOLVER.IMS_PER_BATCH=1 (=N if you run on N GPUs) and SOLVER.TUNING_HIGHLEVEL_OVERRIDE="vision_query" (to only tune the GCPs).

Hello! Thank you for your interest in our work.
I have tried training MQ-Grounding-DINO using a batch size of 1, and it requires approximately 9.6GiB of RAM per GPU.

In my case, it required more than 11g of gpu memory and show up that OutOfMemoryError. I already set the batch size as 1, and use the tiny backbone of the g-dino. And also the optimizer I change to sgd to make the training using less gpu memory. But still it doesn't work. Is there any way I can reduce the usage of gpu memory?

Hi,I encountered the same problem,and my GPU is 2080ti too.Did you solve the problem?
Appreciate for your reply!


WIP: Extract typed schema

Related to:
https://github.com/fastify/fluent-schema/issues/22

[x] BaseSchema (id, title, example, required, etc.)
[x] StringSchema (format, pattern, etc)
[x] ArraySchema (items, etc)
[x] ObjectSchema (properties, etc)
[x] NumberSchema (min, max, etc)
[x] IntegerSchema (min, max, etc)
[x] BooleanSchema
[x] NullSchema
[x] MixedSchema (type accepts an array of types) 🤔
[x] Refactor Typescript types
[x] fix this one as well: https://github.com/fastify/fluent-schema/issues/23

Most of the old Syntax works out of the box however isn't possible anymore chain everything.
e.g.
 FluentSchema()
            .prop('bar').asString().pattern(/.*/g)
            .prop('foo')
            .valueOf().properties.value.type

I need to test much more as well a couple of FIXME still in the code.

[x] required can accept an array FluentSchema().required(['foo', 'bar'])
[x] Support this combining schema (https://json-schema.org/understanding-json-schema/reference/array.html):

{
  "definitions": {
    "address": {
      "type": "object",
      "properties": {
        "street_address": { "type": "string" },
        "city":           { "type": "string" },
        "state":          { "type": "string" }
      },
      "required": ["street_address", "city", "state"]
    }
  },

  "allOf": [
    { "$ref": "#/definitions/address" },
    { "properties": {
        "type": { "enum": [ "residential", "business" ] }
      }
    }
  ]
}

[x] Support this combining schema:

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "$id": "http://foo",
  "type": "object",
  "title": "title",
  "required": ["foo", "bar"],
  "properties": {
    "foo": {
      "type": "boolean",
      "default": "false"
    },
    "thenFooA": {
      "type": "string"
    },
    "thenFooB": {
      "type": "string"
    }
  },
  "allOf": [
    {
      "if": {
        "properties": {
          "foo": {
            "enum": [false]
          }
        }
      },
      "then": {
        "required": ["thenFooA", "thenFooB"]
      }
    },
    {
      "if": {
        "properties": {
          "bar": {
            "enum": ["BAR"]
          }
        }
      },
      "then": {
        "required": ["thenBarA", "thenBarB"]
      }
    }
  ]
}

+1, but it’s < 1.0.0, and this is a breaking chsnge anyway
Il giorno mer 2 gen 2019 alle 20:04 Lorenzo Sicilia <
<EMAIL_ADDRESS>ha scritto:

@aboutlo commented on this pull request.
In src/FluentSchema.js
https://github.com/fastify/fluent-schema/pull/24#discussion_r244822679:

return FluentSchema({

 schema: {

   ...schema,

   if: patchIdsWithParentId({

     schema: ifClauseSchema,

     ...options,

     parentId: '#if',

   }),

   then: patchIdsWithParentId({

     schema: thenClauseSchema,

     ...options,

     parentId: '#then',

   }),

 },

asArray: () =>

But it would break the API. I could create the new syntax and deprecate
the old one.
—
You are receiving this because your review was requested.
Reply to this email directly, view it on GitHub
https://github.com/fastify/fluent-schema/pull/24#discussion_r244822679,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AADL443rCj0xFpiOCK-vgUytCrc3wWPSks5u_QKggaJpZM4Zm5WN
.

+1, but it’s < 1.0.0, and this is a breaking chsnge anyway

Yeah, I was too optimistic to keep the old API valid. I just forced to use  FluentSchema.asObject() always in order to handle the composition keywords correctly. https://github.com/fastify/fluent-schema/pull/24/commits/8bad1a79f72f856c7cc065576b409e021995c459

I don’t understand why “asObject” instead of just “object”, the “as” in English is not correct.

I meant, I need to mark the FluentSchema as Object because now by default
it returns a BaseSchema not an Object. Once I get back all the tests green
I will refactor to use FluentSchema.object()
On Thu, 3 Jan 2019 at 00:08, Matteo Collina<EMAIL_ADDRESS>wrote:

I don’t understand why “asObject” instead of just “object”, the “as” in
English is not correct.
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
https://github.com/fastify/fluent-schema/pull/24#issuecomment-451024052,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABLdLC50dpwfTfISEFJJL4UBfq7jqSeKks5u_UnwgaJpZM4Zm5WN
.

--
Sent from mobile phone

actually, no I guess I misread the report :)

Do you think it could be possible to avoid the call to FluentSchema  like so FluentSchema.object()?

good point, it wasn't like that at the very begining. I could by default return an ObjectSchema, but then I don't like too mych combining schemas (e.g. anyOf, allOf, etc.). Those keywords are not related to a type so I need a BaseSchema aka no type.
Current
const schema = FluentSchema()
      .object()
      .prop('foo', FluentSchema().anyOf([FluentSchema().string()]))
      .valueOf()

pros

FluentSchema return a BaseSchema so combining schema it's nicer

cons

Basic usage isn't straightforward

Proposals

const schema = FluentSchema()
      .prop('foo', FluentSchema().raw().anyOf([FluentSchema().string()]))
      .valueOf()

const schema = FluentSchema()
      .prop('foo', FluentSchema().base().anyOf([FluentSchema().string()]))
      .valueOf()

const schema = FluentSchema()
      .prop('foo', FluentSchema().compose().anyOf([FluentSchema().string()]))
      .valueOf()

All the three proposals have:
pros

Basic usage is easier

cons

Combining schemas require

I’m sorry I could have been clearer. I mean removing the call to FluentSchema, not object(). From FluentSchema().object() to FluentSchema.object().

I see. Currently FluentSchema accepts options e.g. FluentSchema({generateIds:true}) in the future I can see this part expading e.g. FluentSchema({generateIds:true, draft: 'v9', customFormats: [myFormat]}).  So it doesn't seem possible to me.

I see. Currently FluentSchema accepts options e.g. FluentSchema({generateIds:true}) in the future I can see this part expading e.g. FluentSchema({generateIds:true, draft: 'v9', customFormats: [myFormat]}).  So it doesn't seem possible to me.

I think it's possible.
Something like:
module.exports = {
  FluentSchema,
  FORMATS,
  default: FluentSchema,
  object () {
    return new FluentSchema().object()
  }
  // ... all types
}

So it could be used:
const S = require('fluent-schema')
const schema = S.object()
      .prop('foo', S.anyOf([S.string()]))
      .valueOf()

This is additive to the current syntax, it just applies some defaults.

I think the single letter is slightly less verbose.
It might be great if we could get a three chars npm package: https://www.npmjs.com/package/fsa.

Use FS would be nice as well but it collides with the FileSystem API :/
The NPM package is a nice to have :)

Before changing the API I want to finalize the current implementation.
I found that type can accept multiple types 🙌
https://json-schema.org/understanding-json-schema/reference/type.html
https://cswr.github.io/JsonSchema/spec/multiple_types/
I could add something like:
FluentSchema()
  .mixed(FluentSchema().string(), FluentSchema().number())
  .maximum(100)
  .minLength(5)

But I have to dig into Typescript types 😎

I think we should:

land this as a semver-minor (0.6.0)
investigate how to make fastify work with this (I think it's a fastify issue mainly)

Ok, I'm fine with 0.6.0 so we can understand the Fastify issue as well. I will do that this evening hopefully

@Eomm I guess you have an issue here:
 .prop(
    'hello',
    'first#'
  )

it should be:
 .prop('hello').ref('#first')

@aboutlo yeah, I was trying to generate a final "non standard" json-schema like this:
{
  $id: 'second',
  type: 'object',
  allOf: [
    {
      type: 'object',
      properties: {
        second: {
          type: 'number'
        }
      }
    },
    'first#'
  ]
}

But, as Matteo said, it would be a forced feature and we will find a way in the fastify code base 👍
Thank you for your patience and the really good job 💪


Buggy BioPerl warning too verbose

The check for the Bioperl revcom issue is very useful but the logging is far too verbose. When I run AliTV I get one of these messages (depending on the machine):
INFO - MAF input file detected, but Bioperl is bugfree... Therefore, workaround for revcom issue is not activated

or
INFO - MAF input file and buggy BioPerl detected... Therefore, workaround for revcom issue activated

repeated many many times, poluting the output. This message printed once (and not once per maf import) would be sufficient as the BioPerl version does not change during a run.

It also tends to confuse users e.g. https://github.com/AliTVTeam/AliTV/issues/163

I totally agree! Will have a look on the issue on the weekend.


Github workflow to package and release extension

Right now, we locally build the extension package and upload it to vscode marketplace. See if it can be automated fully on releases -- at the very least, we can package the extension and have it be available for download on the releases page.
Current packaging instructtions:
https://github.com/wayfair-tremor/tremor-vscode/blob/master/development.md#packaging

@anupdhml @Licenser I would like to work on this 👀


Update manifests

This pull request includes:

[ ] adequate testing for the new functionality or fixed issue
[ ] adequate documentation informing people about the change such as

[ ] submit a PR for the guides repository if this PR changed any behavior described there: https://www.osbuild.org/guides/

FTR azure & aws test jobs failed on cloud-image-val. Nico is working on this so we can either restart these jobs later and/or admin merge !


InvalidServerResponse

Since last two days, InvalidServerResponse error gets displayed every single time.
No code changes and this thing was working before.

@Bishalsingh can you provide more info? How is the object you're sending? any especial character?

@Bishalsingh I had got the same issue one hour ago. I just added one line with console.log to the library code and was able to see the actual reason of the error. In my case the error was caused by wrong Server key. I added redundant quotes around the key. Once I removed them everything became working properly.

Note that your server key might have changed in your firebase console. it happened to one of my projects once.

You can recheck your key at https://console.firebase.google.com/project/_/settings/cloudmessaging

I am having the same issue.
I already checked my key and it's fine.

Same issue, key is fine if I fire manually via curl.

@wuestkamp indeed, the key to was meant to be used in messages to ONE SINGLE device. For multicast messages, we need to use the key registration_ids. If i get some free time these days I can make an implementation to check these cases and informe more clearly than FCM itself

change key of inner variable of data.. if u use (message_type) as key ... server will send InvalidServerResponse

@1211amarsingh I too facing same issue your message helped me a lot. thank you so much

@jlcvp did you find any solution of this

@varun22mittal I'm not maintaining this lib anymore, I strongly recommend you to use the firebase's official node sdk


solarYield recordings with RC300 results in kWh instead of Wh

Problem
After installing HA 2023.3.5 the energy dashboard shows the recording values of the solarYield in kWh instead of Wh.
To Reproduce
Steps to reproduce the behavior:

Use new statistic API
Take a look into an energy recording using "unitOfMeasure": "Wh"
Values shown in unit kWh instead of expected Wh.

Expected behavior
Values with unit Wh are shown in Wh.
Screenshots

Version

HA version 2023.3.5
HA Bosch component version v0.21.0
Python Bosch HTTP client version?

** Debug SCAN **
bosch_scan.json.txt
Additional context
The data base entry for the statistic shows NULL, expected are Wh.

I removed the high peak and with v0.22.0dev1 the general graph looks pretty good, but now all the values has kWh intead of Wh.

I have enabled the debug severity level and added also an additional log Information on recording.py line 78:
_LOGGER.debug("Update of sensor %s unit: bosch = %s rec = %s", self.unique_id, self._bosch_object.unit_of_measurement, self._unit_of_measurement)
In the log
error_log.txt
it's shown that the unit for the solarYield is added with none instead of Wh:
2023-03-24 05:17:59.471 DEBUG (MainThread) [custom_components.bosch.sensor.recording] Update of sensor RecordingRecording rsolarYield917361676 unit: bosch = None rec = None

Hi, I have installed version 0.22.2 and still the solaryield has no unit:

home-assistant_2023-04-05T17-51-48.697Z.log

Are you able to install bosch_cli on your PC? I'd need result of single day query

Hi pszafer,
I tried to use bosch-thermostat-client-python-0.22.2,
but there are version conflicts visible when installing the requirements.txt:
The user requested sortedcollections==2.1.0
aioxmpp 0.13.1 depends on sortedcollections~=1.0

In the meantime I was able to run the query:
bosch_cli query --config ./config.yml --debug --path /solarCircuits/sc1/solarYield --path /heatSources/actualCHPower --path /heatSources/actualDHWPower --path /heatSources/actualPower

results in
[
    {
        "id": "/solarCircuits/sc1/solarYield",
        "recordable": 1,
        "type": "floatValue",
        "unitOfMeasure": "Wh",
        "value": 143.2,
        "writeable": 0
    },
    {
        "id": "/heatSources/actualCHPower",
        "recordable": 1,
        "type": "floatValue",
        "unitOfMeasure": "kW",
        "value": 0.0,
        "writeable": 0
    },
    {
        "id": "/heatSources/actualDHWPower",
        "recordable": 1,
        "type": "floatValue",
        "unitOfMeasure": "kW",
        "value": 0.0,
        "writeable": 0
    },
    {
        "id": "/heatSources/actualPower",
        "recordable": 1,
        "type": "floatValue",
        "unitOfMeasure": "kW",
        "value": 0.0,
        "writeable": 0
    }
]

Those values should be available as sensors in HA already, right?

Yes, each of the listed sensors is available as a recording sensor (I switched on the new recording API).

All of these 4 recording sensors are available with values in the statistic db
All 3 actual*Power sensors having the expected units (kWh).
The only problem is that the values of the solarYield have no unit and homeassistant chooses then kWh instead of Wh in the energy monitor.

Fixed in https://github.com/bosch-thermostat/home-assistant-bosch-custom-component/releases/tag/v0.23.1


Replace error messages and highlight by output channel

Using the error message to display compilation error is not ergonomic. When error is to long or worst multiple, it is difficult to read it in the error message popup. Then it may be difficult to associate an error and an highlighted place.
Output channels are made for that purpose. When using them to display an error output from a compiler, that displays for each error a filename, a line number and a character position, that part becomes clickable to reach the error place.
Therefore the highlight decoration are no longer needed. Not only UX is better but code is simpler.

Available in V1.1.0


Does not support authorization code grant type?

Hi,  How to execute an Authorization Code Grant Flow by this ?

You would use this library to convert the authorization code returned from the authorization site to an access/refresh token.
e.g.
    private static final String GRANT_TYPE = "authorization_code";
...
        Map<String, String> map = new HashMap<>();
        map.put("code", code);
        map.put("redirect_uri", REDIRECT_URL);

        OAuth2Client client = new OAuth2Client.Builder(GOOGLE_LOGIN_CLIENT_ID,
                GOOGLE_LOGIN_CLIENT_SECRET,
                OAUTH_URL)
                .grantType(GRANT_TYPE)
                .parameters(map)
                .build();

        client.requestAccessToken();


Demonstrate Different Configurations of Triples for Analysis

Include a whole section for this, showing and describing the different representations of triples.

By default, we lemmatize the verb captured in leftward movement:
While undergoing this part of their sentence , they were subjected to a moral discipline to prepare them for the subsequent portions of it
they-undergoing-sentence -> they-undergo-sentence

In considering the measure, he thought it desirable to draw a broad line of distinction between that part of it which related to the Law of Settlement and Bastardy, and which he would at present abstain from discussing, and the changes which it proposed in the Law of Relief, and the mode in which it was to be administered.
he-considering-measure -> he-consider-measure


Log errors to sentry if enabled in deep log

:white_check_mark: coverage increased from 0% to 100%


Update Links in the Service Standard Pages

Why
We should look to add additional further information from more recent blogs as this not only gives additional relevant recent information to users but ensure users are confident the service manual is being kept up to date.
What
All guidance links are dated – all information is 2 years old.
GDS have not been updating guidance generally over the past 12-18 months – this is likely to change with new team.
Done When

[x] Spoken to UCD team for insights on blogs/latest work
[ ] All points are reviewed
[ ] Links updated

To Do

[x] Point 1:

Links are not all up to date, are there more recent posts from Matt Edgar that are relevant?

[x] Points 2 and 3

Encouraging people to use your service online – consider something about not being digital first but still want to maximise the potential usage of the digital service.
The more recent blogs by Matt Edgar may be relevant here.

[ ] Point 10

Some of the links would benefit from more recent information being found.

[ ] Point 11

Are there any policy papers from NHSx that should be linked form this section

[ ] Point 12

Links are dated but may be the latest versions as the idea of coding in the open has been around for a relative long time (All links seem to be from 2017).

Confluence page with updated links including service standard points 4 & 5

@georginaplatt, shall we have a chat this week?

@georginaplatt, I've created a task for myself #1390 . Shall we keep this issue open till you get a response from Shan and Terrence?


Use spark join to map dict on DataFrameWrapper.data

Addresses #205.
Converts lookup dictionary to spark dataframe which can be joined to .data and the values extracted.
~259x faster (than previous create_map usage) for 3_000_000 rows/unique categories.

Thanks for submitting this PR, it's looking good. I just gave it a spin on Databricks with the following dataset:
from sds_data_model.dataframe import DataFrameWrapper

phi_sdf = DataFrameWrapper.from_files(name = 'pri_hab',
    data_path = '/dbfs/mnt/base/unrestricted/source_defra_data_services_platform',
    metadata_path = 'https://ckan.publishing.service.gov.uk/harvest/object/85e03bf0-4e95-4739-a5fa-21d60cf7f069',
    read_file_kwargs = {'pattern': 'dataset_priority_habitat_inventory_*/format_SHP_priority_habitat_inventory_*/LATEST_priority_habitat_inventory_*/PHI_v2_3_*',
    'suffix': '.shp',
    'ideal_chunk_size':1000})

First I tested a single column for categorisation:
phi_sdf.categorize(['Main_Habit'])

And the output looked as I would expect:
phi_sdf.lookup

Out[4]: {'Main_Habit': {'Deciduous woodland': 0,
  'Fragmented heath': 1,
  'Coastal sand dunes': 2,
  'Saline lagoons': 3,
  'Coastal vegetated shingle': 4,
  'Purple moor grass and rush pastures': 5,
  'Lowland dry acid grassland': 6,
  'Lowland meadows': 7,
  'Lowland fens': 8,
  'Coastal saltmarsh': 9,
  'Grass moorland': 10,
  'Limestone pavement': 11,
  'Lowland calcareous grassland': 12,
  'Mountain heaths and willow scrub': 13,
  'Good quality semi-improved grassland': 14,
  'Lowland raised bog': 15,
  'Upland heathland': 16,
  'Reedbeds': 17,
  'No main habitat but additional habitats present': 18,
  'Maritime cliff and slope': 19,
  'Lowland heathland': 20,
  'Upland calcareous grassland': 21,
  'Calaminarian grassland': 22,
  'Blanket bog': 23,
  'Traditional orchard': 24,
  'Upland hay meadow': 25,
  'Mudflats': 26,
  'Upland flushes, fens and swamps': 27,
  'Coastal and floodplain grazing marsh': 28}}

I then thought I would try with multiple columns:
phi_sdf.categorize(['Main_Habit','Source1'])

And I get a different output, but the Main_Habit names have been replaced with integers, and in a strange order, yet the Source1 names are present in the lookup:
phi_sdf.lookup

Out[10]: {'Main_Habit': {26: 0,
  19: 1,
  0: 2,
  22: 3,
  7: 4,
  25: 5,
  6: 6,
  9: 7,
  27: 8,
  17: 9,
  28: 10,
  5: 11,
  1: 12,
  10: 13,
  3: 14,
  12: 15,
  8: 16,
  11: 17,
  2: 18,
  4: 19,
  13: 20,
  18: 21,
  14: 22,
  21: 23,
  15: 24,
  23: 25,
  20: 26,
  16: 27,
  24: 28},
 'Source1': {'English Nature Lowland Grassland Inventory': 0,
  'North Pennines AONB Partnership 2010': 1,
  'NZ0384': 2,
  'SE3860': 3,
  'SK8064': 4,
  'SE4317': 5,
  'SK4534': 6,
  'Engish Nature Humber to Pennines Team SSSI NVC Survey': 7,
  'Naddle Forest SSSI Habitat Survey': 8,
  'Hulme Park Phase 1': 9,
  'Waldridge Fell Habitat Survey map': 10,
  'SE4655': 11,
  'SE9302': 12,
  'The SBI Resurvey of East Staffordshire District': 13,
  'Derbyshire Calaminarian Grassland Project 2011': 14,
  'Trent Washlands Survey 2006': 15,

I see this as a problem with either:
a) Overwriting an existing lookup with a new lookup which contains the same column
b) An issue with providing multiple columns to categorize.
I guess my question is, does the method handle overwriting the lookup stored within DataFrameWrapper? If so, is it handling multiple columns as expected?
Do you have any thoughts/ideas?

And I get a different output, but the Main_Habit names have been replaced with integers, and in a strange order, yet the Source1 names are present in the lookup

@JamesKeny0n can probably explain this one slightly better, but I found a similar thing when testing. In discuss with James, it appeared to result from some sort of larger issue with retained states. Basically, a work around would be to detach and reattach before trying to categorize again, and you should then see what you expected.

The issue is that categorize  was called once on "Main_Habit" and  it worked, the underlying strings were mapped to ints. But you called categorize again on "Main_Habit", now you are mapping the underlying ints to new ints.
categorize works inplace.

Thanks @danlewis85  and @JamesKeny0n. I've added a bit of detail to the categorize docstring and am happy to merge this branch once checks are satisfied.


missing -lthrift flag when linking R arrow.so

Describe the bug, including details regarding any error messages, version, and platform.
Hi! I have compiled C++ libarrow from source, and installed it under my home directory. I am trying to install arrow R package from source, and I expected that I should be able to do that without manually adding any linker flags. However, I observe that the linker step creates arrow.so with libthrift link not found, unless I add LDFLAGS=-lthrift in my ~/.R/Makevars file (which R reads to add flags to the linker command). Is this a bug? Does -lthrift need to be added to some config file that determines what flags are used for building the R package? Probably arrow/r/configure needs to generate arrow/r/src/Makevars with -lthrift under PKG_LIBS, which it does not have on my system, see below:
PKG_CPPFLAGS=-I/home/tdhock/include  -DARROW_R_WITH_PARQUET -DARROW_R_WITH_DATASET -DARROW_R_WITH_ACERO -DARROW_R_WITH_JSON
CXX_STD=CXX17
PKG_LIBS=-L/home/tdhock/lib -larrow_acero -larrow_dataset -lparquet -larrow

First, with LDFLAGS=-L${HOME}/lib -Wl,-rpath=${HOME}/lib -L${CONDA_PREFIX}/lib -Wl,-rpath=${CONDA_PREFIX}/lib -lthrift in ~/.R/Makevars it works as shown below
(arrow) tdhock@maude-MacBookPro:~/arrow-git/cpp/build(main*)$ rm ../../r/src/arrow.so && LDFLAGS="-L$HOME/lib -Wl,-rpath=$HOME/lib -L$CONDA_PREFIX/lib -Wl,-rpath=$CONDA_PREFIX/lib -lthrift" ARROW_DEPENDENCY_SOURCE=SYSTEM ARROW_R_DEV=true LIBARROW_BINARY=false PKG_CONFIG_PATH=$HOME/lib/pkgconfig:$CONDA_PREFIX/lib/pkgconfig R CMD INSTALL ../../r
Loading required package: grDevices
* installing to library ‘/home/tdhock/lib/R/library’
* installing *source* package ‘arrow’ ...
** using staged installation
*** Generating code with data-raw/codegen.R
Loading required package: grDevices
Error in library(decor) : there is no package called ‘decor’
Calls: suppressPackageStartupMessages -> withCallingHandlers -> library
Execution halted
*** Trying Arrow C++ found by pkg-config: /home/tdhock
*** > Packages are both on development versions (13.0.0-SNAPSHOT, <IP_ADDRESS>00)
*** > If installation fails, rebuild the C++ library to match the R version
*** > or retry with FORCE_BUNDLED_BUILD=true
PKG_CFLAGS=-I/home/tdhock/include  -DARROW_R_WITH_PARQUET -DARROW_R_WITH_DATASET -DARROW_R_WITH_ACERO -DARROW_R_WITH_JSON
PKG_LIBS=-L/home/tdhock/lib -larrow_acero -larrow_dataset -lparquet -larrow
** libs
using C++ compiler: ‘g++ (GCC) 10.1.0’
using C++17
g++ -std=gnu++17 -shared -L/home/tdhock/lib/R/lib -L/home/tdhock/lib -Wl,-rpath=/home/tdhock/lib -L/home/tdhock/.local/share/r-miniconda/envs/arrow/lib -Wl,-rpath=/home/tdhock/.local/share/r-miniconda/envs/arrow/lib -lthrift -o arrow.so RTasks.o altrep.o array.o array_to_vector.o arraydata.o arrowExports.o bridge.o buffer.o chunkedarray.o compression.o compute-exec.o compute.o config.o csv.o dataset.o datatype.o expression.o extension-impl.o feather.o field.o filesystem.o io.o json.o memorypool.o message.o parquet.o r_to_arrow.o recordbatch.o recordbatchreader.o recordbatchwriter.o safe-call-into-r-impl.o scalar.o schema.o symbols.o table.o threadpool.o type_infer.o -L/home/tdhock/lib -larrow_acero -larrow_dataset -lparquet -larrow -L/home/tdhock/lib/R/lib -lR
/usr/bin/ld: warning: /home/tdhock/.local/share/r-miniconda/envs/arrow/lib/libstdc++.so: unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010001
/usr/bin/ld: warning: /home/tdhock/.local/share/r-miniconda/envs/arrow/lib/libstdc++.so: unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010002
/usr/bin/ld: warning: /home/tdhock/.local/share/r-miniconda/envs/arrow/lib/libgcc_s.so.1: unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010001
/usr/bin/ld: warning: /home/tdhock/.local/share/r-miniconda/envs/arrow/lib/libgcc_s.so.1: unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010002
/usr/bin/ld: warning: /home/tdhock/.local/share/r-miniconda/envs/arrow/lib/libgcc_s.so.1: unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010001
/usr/bin/ld: warning: /home/tdhock/.local/share/r-miniconda/envs/arrow/lib/libgcc_s.so.1: unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010002
installing to /home/tdhock/lib/R/library/00LOCK-r/00new/arrow/libs
** R
** inst
** byte-compile and prepare package for lazy loading
Loading required package: grDevices
** help
*** installing help indices
** building package indices
Loading required package: grDevices
** installing vignettes
** testing if installed package can be loaded from temporary location
Loading required package: grDevices
** checking absolute paths in shared objects and dynamic libraries
** testing if installed package can be loaded from final location
Loading required package: grDevices
** testing if installed package keeps a record of temporary installation path
* DONE (arrow)

Second, with LDFLAGS=-L${HOME}/lib -Wl,-rpath=${HOME}/lib -L${CONDA_PREFIX}/lib -Wl,-rpath=${CONDA_PREFIX}/lib I get a broken link shown below,
(arrow) tdhock@maude-MacBookPro:~/arrow-git/cpp/build(main*)$ rm ../../r/src/arrow.so && LDFLAGS="-L$HOME/lib -Wl,-rpath=$HOME/lib -L$CONDA_PREFIX/lib -Wl,-rpath=$CONDA_PREFIX/lib -lthrift" ARROW_DEPENDENCY_SOURCE=SYSTEM ARROW_R_DEV=true LIBARROW_BINARY=false PKG_CONFIG_PATH=$HOME/lib/pkgconfig:$CONDA_PREFIX/lib/pkgconfig R CMD INSTALL ../../r
Loading required package: grDevices
* installing to library ‘/home/tdhock/lib/R/library’
* installing *source* package ‘arrow’ ...
** using staged installation
*** Generating code with data-raw/codegen.R
Loading required package: grDevices
Error in library(decor) : there is no package called ‘decor’
Calls: suppressPackageStartupMessages -> withCallingHandlers -> library
Execution halted
*** Trying Arrow C++ found by pkg-config: /home/tdhock
*** > Packages are both on development versions (13.0.0-SNAPSHOT, <IP_ADDRESS>00)
*** > If installation fails, rebuild the C++ library to match the R version
*** > or retry with FORCE_BUNDLED_BUILD=true
PKG_CFLAGS=-I/home/tdhock/include  -DARROW_R_WITH_PARQUET -DARROW_R_WITH_DATASET -DARROW_R_WITH_ACERO -DARROW_R_WITH_JSON
PKG_LIBS=-L/home/tdhock/lib -larrow_acero -larrow_dataset -lparquet -larrow
** libs
using C++ compiler: ‘g++ (GCC) 10.1.0’
using C++17
g++ -std=gnu++17 -shared -L/home/tdhock/lib/R/lib -L/home/tdhock/lib -Wl,-rpath=/home/tdhock/lib -L/home/tdhock/.local/share/r-miniconda/envs/arrow/lib -Wl,-rpath=/home/tdhock/.local/share/r-miniconda/envs/arrow/lib -o arrow.so RTasks.o altrep.o array.o array_to_vector.o arraydata.o arrowExports.o bridge.o buffer.o chunkedarray.o compression.o compute-exec.o compute.o config.o csv.o dataset.o datatype.o expression.o extension-impl.o feather.o field.o filesystem.o io.o json.o memorypool.o message.o parquet.o r_to_arrow.o recordbatch.o recordbatchreader.o recordbatchwriter.o safe-call-into-r-impl.o scalar.o schema.o symbols.o table.o threadpool.o type_infer.o -L/home/tdhock/lib -larrow_acero -larrow_dataset -lparquet -larrow -L/home/tdhock/lib/R/lib -lR
/usr/bin/ld: warning: /home/tdhock/.local/share/r-miniconda/envs/arrow/lib/libstdc++.so: unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010001
/usr/bin/ld: warning: /home/tdhock/.local/share/r-miniconda/envs/arrow/lib/libstdc++.so: unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010002
/usr/bin/ld: warning: /home/tdhock/.local/share/r-miniconda/envs/arrow/lib/libgcc_s.so.1: unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010001
/usr/bin/ld: warning: /home/tdhock/.local/share/r-miniconda/envs/arrow/lib/libgcc_s.so.1: unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010002
/usr/bin/ld: warning: /home/tdhock/.local/share/r-miniconda/envs/arrow/lib/libgcc_s.so.1: unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010001
/usr/bin/ld: warning: /home/tdhock/.local/share/r-miniconda/envs/arrow/lib/libgcc_s.so.1: unsupported GNU_PROPERTY_TYPE (5) type: 0xc0010002
installing to /home/tdhock/lib/R/library/00LOCK-r/00new/arrow/libs
** R
** inst
** byte-compile and prepare package for lazy loading
Loading required package: grDevices
** help
*** installing help indices
** building package indices
Loading required package: grDevices
** installing vignettes
** testing if installed package can be loaded from temporary location
Loading required package: grDevices
Error: package or namespace load failed for ‘arrow’ in dyn.load(file, DLLpath = DLLpath, ...):
 unable to load shared object '/home/tdhock/lib/R/library/00LOCK-r/00new/arrow/libs/arrow.so':
  libthrift.so.0.15.0: cannot open shared object file: No such file or directory
Error: loading failed
Execution halted
ERROR: loading failed
* removing ‘/home/tdhock/lib/R/library/arrow’
* restoring previous ‘/home/tdhock/lib/R/library/arrow’
(arrow) tdhock@maude-MacBookPro:~/arrow-git/cpp/build(main*)$ ldd ../../r/src/arrow.so
	linux-vdso.so.1 (0x00007ffc9e8c4000)
	libgtk3-nocsd.so.0 => /usr/lib/x86_64-linux-gnu/libgtk3-nocsd.so.0 (0x00007f03ee2cb000)
	libarrow_acero.so.1300 => /home/tdhock/lib/libarrow_acero.so.1300 (0x00007f03eda3f000)
	libarrow_dataset.so.1300 => /home/tdhock/lib/libarrow_dataset.so.1300 (0x00007f03ecfc8000)
	libparquet.so.1300 => /home/tdhock/lib/libparquet.so.1300 (0x00007f03ec5c2000)
	libarrow.so.1300 => /home/tdhock/lib/libarrow.so.1300 (0x00007f03e899d000)
	libR.so => /usr/lib/libR.so (0x00007f03e8374000)
	libstdc++.so.6 => /home/tdhock/.local/share/r-miniconda/envs/arrow/lib/libstdc++.so.6 (0x00007f03e8160000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f03e7dc2000)
	libgcc_s.so.1 => /home/tdhock/.local/share/r-miniconda/envs/arrow/lib/libgcc_s.so.1 (0x00007f03eebd5000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f03e79d1000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f03e77cd000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f03e75ae000)
	librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f03e73a6000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f03ee9f8000)
	libthrift.so.0.15.0 => not found
	libssl.so.1.1 => /usr/lib/x86_64-linux-gnu/libssl.so.1.1 (0x00007f03e7119000)
	libcrypto.so.1.1 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007f03e6c4d000)
	libblas.so.3 => /usr/lib/x86_64-linux-gnu/libblas.so.3 (0x00007f03e69e0000)
	libreadline.so.7 => /lib/x86_64-linux-gnu/libreadline.so.7 (0x00007f03e6797000)
	libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3 (0x00007f03e6526000)
	liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f03e6300000)
	libbz2.so.1.0 => /lib/x86_64-linux-gnu/libbz2.so.1.0 (0x00007f03e60f0000)
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f03e5ed3000)
	libicuuc.so.60 => /usr/lib/x86_64-linux-gnu/libicuuc.so.60 (0x00007f03e5b1b000)
	libicui18n.so.60 => /usr/lib/x86_64-linux-gnu/libicui18n.so.60 (0x00007f03e567a000)
	libgomp.so.1 => /usr/lib/x86_64-linux-gnu/libgomp.so.1 (0x00007f03e5437000)
	libtinfo.so.5 => /lib/x86_64-linux-gnu/libtinfo.so.5 (0x00007f03e520d000)
	libicudata.so.60 => /usr/lib/x86_64-linux-gnu/libicudata.so.60 (0x00007f03e3664000)
(arrow) tdhock@maude-MacBookPro:~/arrow-git/cpp/build(main*)$

This is with arrow from git, on Ubuntu 18.04, old intel 64-bit CPU.
Component(s)
R

We don't need -lthrift build flag.
We need the LD_LIBRARY_PATH=${CONDA_PREFIX}/lib:${LD_LIBRARY_PATH} environment variable on run-time: LD_LIBRARY_PATH=${CONDA_PREFIX}/lib:${LD_LIBRARY_PATH} ... R CMD INSTALL ...
(I assume that your libtrift.so is installed by conda.)

Of course setting LD_LIBRARY_PATH works, but I expected that I should be able to build the R package and then run it without having to set LD_LIBRARY_PATH. If that is not something you would like to support you can close this.

cmake -DCMAKE_INSTALL_RPATH=${CONDA_PREFIX}/lib ... may help you.
BTW, why do you need to specify conda related paths explicitly? I think that conda activate or something sets related environments such as LD_LIBRARY_PATH and PKG_CONFIG_PATH automatically.

I think that conda activate or something sets related environments such as LD_LIBRARY_PATH and PKG_CONFIG_PATH automatically.

That's not quite accurate.  conda-build configures the origin / loader_path of shared libraries:
Relative links require a special variable in the link itself:

    On Linux, the $ORIGIN variable allows you to specify "relative to this file as it is being executed".

    On macOS, the variables are:

        @rpath---Allows you to set relative links from the system load paths.

        @loader_path---Equivalent to $ORIGIN.

        @executable_path---Supports the Apple .app directory approach, where libraries know where they live relative to their calling application.

Conda-build uses @loader_path on macOS and $ORIGIN on Linux because we install into a common root directory and can assume that other libraries are also installed into that root. The use of the variables allows you to build relocatable binaries that can be built on one system and sent everywhere.

On Linux, conda-build modifies any shared libraries or generated executables to use a relative dynamic link by calling the patchelf tool. On macOS, the install_name_tool tool is used.

However, that is the responsibility of conda-build and not the package being built (e.g. building arrow directly shouldn't configure the rpath but conda-forge's arrow recipe should).  This does mean, if you are building and installing into conda directly by setting CMAKE_INSTALL_PREFIX (I do this myself), then you either need to set the rpath manually (to emulate conda-build) or set LD_LIBRARY_PATH.

Above I was installing under my home directory, and linking to thrift from a conda env,
installing to /home/tdhock/lib/R/library/00LOCK-r/00new/arrow/libs
,,,
g++ -shared ... -L/home/tdhock/.local/share/r-miniconda/envs/arrow/lib -Wl,-rpath=/home/tdhock/.local/share/r-miniconda/envs/arrow/lib ...

It is strange that this works for the other links, -larrow_acero -larrow_dataset -lparquet -larrow is included in the linker line by default, but -lthrift is missing. I expected that either all the required -l flags should be present, or none. (and user should not have to set LD_LIBRARY_PATH, that is highly unusually when installing R packages)

If you prefer rpath, you need to set rpath to Apache Arrow C++ (not Apache Arrow R) by cmake -DCMAKE_INSTALL_RPATH=${CONDA_PREFIX}/lib ... as mentioned in https://github.com/apache/arrow/issues/35577#issuecomment-1546414064 .

It is strange that this works for the other links, -larrow_acero -larrow_dataset -lparquet -larrow is included in the linker line by default, but -lthrift is missing.

It's not strange. libtrhift.so is used by libparquet.so but it's not used directly by arrow.so (shared library for R, not libarrow.so provided by Apache Arrow C++).
So we don't need -ltrhfit to build arrow.so (shared library for R, not libarrow.so provided by Apache Arrow C++).

you wrote that libthrift.so is used by libparquet.so but it's not used directly by arrow.so (shared library for R) but that is not true according to ldd on my system (see output above, relevant part shown below)
(arrow) tdhock@maude-MacBookPro:~/arrow-git/cpp/build(main*)$ ldd ../../r/src/arrow.so
...
	libthrift.so.0.15.0 => not found

Also, I think there is some confusion between -rpath flags and -lthrift flag.
Actually, I have no problem with the rpath, that is normal that I set it in my ~/.R/Makevars file, because that is how to tell R to look for libraries to link against in non-standard directories, via LDFLAGS=-L${HOME}/lib -Wl,-rpath=${HOME}/lib -L${CONDA_PREFIX}/lib -Wl,-rpath=${CONDA_PREFIX}/lib, it is completely normal/standard to do that when you have C++ libraries installed in non-standard directories. So your suggestion to modify the rpath via cmake -DCMAKE_INSTALL_RPATH=${CONDA_PREFIX}/lib ... I don't think would fix this issue though, because I told the R linker command about my non-standard rpath already via LDFLAGS in ~/.R/Makevars.
My issue is that the -lthrift flag is missing from the linker command line, when creating the R package arrow.so file, so I get a broken link to thrift, and an error when I try to install the R package (without setting LD_LIBRARY_PATH). I believe that since R arrow depends on thrift (even if indirectly through parquet), then it is your responsibility to ensure that your build script creates a shared library with a valid link to thrift, right?

ldd show dependencies recursively. So non-direct dependencies (libthrift.so in this case) are also shown.
If you want to show direct dependencies, you can use readelf: LANG=C readelf --dynamic ../../r/src/arrow.so | grep Shared
In general, you need to specify rpath when you build Apache Arrow C++ not Apache Arrow R.
Could you try installing Apache Arrow C++ with rpath and installing Apache Arrow R without -lthrift?
Installing Apache Arrow R with LDFLAGS=-L${HOME}/lib -Wl,-rpath=${HOME}/lib -L${CONDA_PREFIX}/lib -Wl,-rpath=${CONDA_PREFIX}/lib -lthrift works because libthrift.so is linked to arrow.so (not libarrow.so nor libparquet.so) with rpath. But arrow.so doesn't refer symbols in libthrift.so directly. So the linking isn't needed. It works but it's not a correct approach. (It's OK that you use this approach if you like it. But we don't recommend this approach.)

actually, cmake -DCMAKE_INSTALL_RPATH=${CONDA_PREFIX}/lib ... solved this issue.
when libparquet.so (built by arrow C++ cmake) has a broken link, it is passed on to the arrow.so in the R package.
when libparquet.so has a good link, it is passed onto the arrow.so in the R package,
(base) tdhock@maude-MacBookPro:~/lib$ ldd ~/arrow-git/r/src/arrow.so |grep thrift
	libthrift.so.0.15.0 => /home/tdhock/.local/share/r-miniconda/envs/arrow/lib/libthrift.so.0.15.0 (0x00007f069ea0e000)
(base) tdhock@maude-MacBookPro:~/lib$ ldd ~/arrow-git/r/src/arrow.so |grep thrift
	libthrift.so.0.15.0 => /home/tdhock/.local/share/r-miniconda/envs/arrow/lib/libthrift.so.0.15.0 (0x00007f40652f2000)

sorry for the trouble.

No problem. :-)


nested bundle

importing relation:
a.yml -> b.yml -> c.yml
d.yml -> b.yml

then adding about c.yml, only once. and c.yml is also imported.


s3_data_dir is ignored in some cases

Hi team,
I have no clue what is going on.
My config block is the following:
{{ config(
    materialized='table',
    table_type='hive',
    format='orc',
    partition_by=['project_id', 'event_date']
) }}

with...

in my dbt_profile.yml this is defined:
default:
  target: local
  outputs:
    dev:
      type: athena
      s3_staging_dir: s3://aws-athena-query-results-.../folder_name/
      s3_data_dir: s3://target-bucket-name/dbt_models/
      s3_data_naming: schema_table
      region_name: eu-central-1
      database: awsdatacatalog
      schema: dbt
      work_group: my-workgroup
      threads: 4
    prd:
      type: athena
      s3_staging_dir: s3://aws-athena-query-results-.../folder_name/
      s3_data_dir: s3://target-bucket-name/dbt_models/
      s3_data_naming: schema_table
      region_name: eu-central-1
      database: awsdatacatalog
      schema: dbt
      work_group: my-workgroup
      threads: 4

If I build this in target = dev I get this execution in my logs:
  create table "schema"."table_name"
  with (
    table_type='hive',
    is_external=true,
    external_location='s3://target-bucket-name/dbt_models/schema/table_name',
    format='orc'
  )
  as

with

But if I build it with target = prd I get this:

On model.schema.table_name: -- /* {"app": "dbt", "dbt_version": "1.4.5", "profile_name": "default", "target_name": "prd", "node_id": "model.schema.table_name"} */

  create table "schema"."table_name"
  with (
    table_type='hive',
    is_external=true,
    format='orc'
  )
  as

Suddenly the whole external_location is missing. I tried to rebuild it multiple times.
The workgroups are defined exactly the same.
I tried to set the versions_to_keep to one assuming that somehow somewhere in GLUE the location is saved somehow but I still get the same error.
dev and prd are different AWS environments but both dbt executions are done by the same CI/CD configuration only the target is changed.
I checked the code but I could not find any reason why the s3_table_prefix function shouldn't use the defined s3_data_dir.
Does anyone have any idea what is going on?

Which adapter version are you using?
Do you have different work_group between dev and prod? if yes, it could be possible that prod has a default workgroup location set and that is used instead (available from 1.4.2). Could you please double check?
Notes

versions_to_keep is something used only for table_hive_ha so irrelevant in your case.

Hi!
I am using 1.4.2 and the workgroup location is not set to default:

But yes dev and prd are different workgroups but the setup is the same.

@CommonCrisis Thanks, you have a location "set", I can see it from the screenshot, query result location is set, then in the adapter uses that as default - that's why you see the query above.
So long story short, if a workgroup have a default location, and you do have one, data will be persisted there.
This https://github.com/dbt-athena/dbt-athena/pull/223 might be interesting for you (merged to main today), if you have a workgroup without enforced configuration, the workgroup location will be ignored, you might want to give it a try, maybe it solves your issue.
Alternatively, consider creating a workgroup only for dbt (recommended), without default location. This allows you to isolate queries coming from dbt (vs queries from other users), and you can have nice cloudwatch metrics about other factors (data scans amount and more..).

Ahhh oke I get it.
Would like to try the main branch buuuut there is a little issue with building the egg:
https://github.com/dbt-athena/dbt-athena/pull/225

Yes! it is working with https://github.com/dbt-athena/dbt-athena/pull/223
Nice!

@CommonCrisis thanks for checking. @juliansteger-sc added the implementation to consider the workgroup setting enforcement. In your case seems that your working group didn't have a workgroup enforcement setting.


Thanks for helping me i serached how i can do it but in vanilla or with other plugin is impossible because i use item meta data.


Display EVM Address

This issue already achieved in flow wallet ios dev version1.0.0(10).
The app display evm address in main page and sidebar.


Newcomers - Inactive repository

There is a lot of issues about maintaining this plugin and I found it hard to use in CI and with the last cypress version
TL;DR the maintained version seems to be there: https://github.com/FRSOURCE/cypress-plugin-visual-regression-diff
Updates are regular and it supports ESM, etc...

This is a good plugin with lots of open PRs and low hanging fruit fixes. I hope it can continue.

TL;DR the maintained version seems to be there: https://github.com/FRSOURCE/cypress-plugin-visual-regression-diff

Not quite - that plugin only provides image diffing, but not for snapshots toMatchSnapshot()


upgrade dockerbuild

Status
Ready/In Progress/In Hold (Reason for hold)
Related Content Pull Request
Related PR: link to the PR at demisto/content
Related Issues
Related: link to the issue
Description
A few sentences describing the overall goals of the pull request's commits.

Docker Image Ready - Dev
Docker automatic build at CircleCI has deployed your docker image: devdemisto/dockerbuild:<IP_ADDRESS>462
It is available now on docker hub at: https://hub.docker.com/r/devdemisto/dockerbuild/tags
Get started by pulling the image:
docker pull devdemisto/dockerbuild:<IP_ADDRESS>462

Docker Metadata

Image Size: 854.44 MB

Image ID: sha256:949abffe3795060715a9bef30f6e6e4eb1af455e2affe94bc97a93a846f65c21

Created: 2023-01-05T15:07:32.5379875Z

Arch: linux/amd64

Command: ["/bin/sh"]

Environment:

PATH=/home/circleci/.local/bin:/home/circleci/bin:/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
LANG=C.UTF-8
GPG_KEY=A035C8C19219BA821ECEA86B64E628F8D684696D
PYTHON_VERSION=3.10.1
PYTHON_PIP_VERSION=21.2.4
PYTHON_SETUPTOOLS_VERSION=57.5.0
PYTHON_GET_PIP_URL=https://github.com/pypa/get-pip/raw/3cb8888cc2869620f57d5d2da64da38f516078c7/public/get-pip.py
PYTHON_GET_PIP_SHA256=c518250e91a70d7b20cceb15272209a4ded2a0c263ae5776f129e0d9b5674309
DEBIAN_FRONTEND=noninteractive
DOCKER_IMAGE=devdemisto/dockerbuild:<IP_ADDRESS>462

Labels:

org.opencontainers.image.authors:Demisto<EMAIL_ADDRESS>org.opencontainers.image.revision:5fb7216b1c389a67f58dc2239a93039cbe9b9743
org.opencontainers.image.version:<IP_ADDRESS>462


Angular and RxJS v6 compatibility

Attempting to migrate a project to Angular v6.0.0-rc.2 with RxJS v6.0.0-rc.0 and rxjs-compat installed. Getting the following error on build:
ERROR in ./node_modules/apollo-angular/utils.js Module not found: Error: Can't resolve 'rxjs/operator/observeOn'

I'm going to migrate everything to RxJS and Angular 6 soon.

I'm going to do this after Wednesday but if someone wants to pick it up than it's fine :)

Woot! Can't wait!!!

Work in progress #580

Right now you all can use it by adding rxjs-compat package

An update https://github.com/apollographql/apollo-angular/pull/580#issuecomment-387744110

You can now yarn add apollo-angular@next, later today I'll release a stable version.


Not clear what tests are being here automated

[Enter feedback here]
It is not immediately clear unless you dive into the document that this document is talking of automating Test Studio tests

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 513096fe-dce9-70cc-4891-13ffdf2447ca
Version Independent ID: 29cdb9ce-32e4-2cc5-522b-76587872d8ab
Content: Automate tests with Azure Pipelines using classic editor - Power Apps
Content Source: powerapps-docs/maker/canvas-apps/test-studio-classic-pipeline-editor.md
Service: powerapps
Sub-service: canvas-maker
GitHub Login: @aengusheaney
Microsoft Alias: aheaney

Thank you for reaching out. The topic is under the table of contents listed as "Test an app with Test Studio", after the topics of "Test Studio Overview", and "Working with Test Studio".


cuda stream destory failed after python test passed

Hi, when running python test scripts, even the test are passing, the cuda stream still fails:
python3 test/python/test_operation.py -v TestPythonOperation.test_sum_cpu
test_sum_cpu (__main__.TestPythonOperation) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.002s

OK
WARNING: Logging before InitGoogleLogging() is written to STDERR
F0618 07:06:09.329576 10741 cuda_gpu.cc:48] Check failed: error == cudaSuccess (29 vs. 0)  driver shutting down
*** Check failure stack trace: ***
Aborted (core dumped)

the cuda stream destory failed is resolved in https://github.com/apache/singa/pull/728

ok thanks


next is not a function when listener called without next fn

Description
Describe your issue here.
From an updated view, pressing a button defined in an actions block with an "app.action(actionId, fn)" defined results in
UnknownError: next is not a function
at Object.asCodedError (/Users/jimmurphy/dev/RMTSlack/node_modules/@slack/bolt/dist/errors.js:25:12)
at App.handleError (/Users/jimmurphy/dev/RMTSlack/node_modules/@slack/bolt/dist/App.js:522:43)
at App.processEvent (/Users/jimmurphy/dev/RMTSlack/node_modules/@slack/bolt/dist/App.js:515:25)
The updated view's block section where the button is defined:
"blocks": [
{
"type": "section",
"text": {
"type": "mrkdwn",
"text": "some text here"
}
},
{
"type": "section",
"fields": [
{
"type": "mrkdwn",
"text": "some text here"
}
]
},
{
"type": "actions",
"elements": [
{
"type": "button",
"action_id": "qualify_input",
"text": {
"type": "plain_text",
"emoji": true,
"text": "Qualify"
},
"style": "primary",
"value": "qualify"
},
{
"type": "button",
"text": {
"type": "plain_text",
"emoji": true,
"text": "Cancel"
},
"style": "danger",
"value": "click_me_123"
}
]
}
]
What type of issue is this? (place an x in one of the [ ])

[ x ] bug
[ ] enhancement (feature request)
[ ] question
[ ] documentation related
[ ] example code related
[ ] testing related
[ ] discussion

Requirements (place an x in each of the [ ])

[ x ] I've read and understood the Contributing guidelines and have done my best effort to follow them.
[ x ] I've read and agree to the Code of Conduct.
[  x ] I've searched for any related issues and avoided creating a duplicate issue.

Bug Report
Filling out the following details about bugs will help us solve your issue sooner.
Reproducible in:
package version:  Bolt 3.4.0
node version:  v16.0.0
OS version(s): Mac OS 10.15.7
Steps to reproduce:

Open modal with view
select from static_select
new view added to modal stack
click button on new view
see stack trace above

Expected result:
What you expected to happen
Action handler should get called
Actual result:
What actually happened
nothing, stack trace is thrown and nothing happens with the modal
Attachments:
Logs, screenshots, screencast, sample project, funny gif, etc.

Hi @jmurphy67! I implemented the steps as you have them outlined above and didn't run into any trouble getting the action handler to invoke. Could you provide us more detail about the fn callback that you're passing into app.action()?

The error is happening when the actions button with "action_id" = "qualify_input"  is pressed from the view.  This is the handler that is setup for the button action:
` app.action('qualify_input'), async ({ ack,body,client }) => {
await ack();
await console.log("here")
}`

but it is never reached.
Not sure if this helps but the view is invoked with:
await client.views.push({         trigger_id: body.trigger_id,         view_id: body.view.id,         hash: body.view.hash,         view: views.release_view(payload,body),     })

After reading through the docs it turns out I needed to listen for a constraint object as in
app.action({ action_id: 'select_user', block_id: 'assign_ticket' }... , which worked.
Thanks.


targets/basic_routing

When the first installation,I follow the README.
cd p4factory/targets/basic_routing/
make bm
p4c_bm/shell.py", line 20, in
from p4_hlir.main import HLIR
ImportError:No module named p4_hlir.main
make:***[bm_tenjin_output_oldest]
I am sure "p4_hlir" have been cloned in submodules successfully .
could you give me some instructions? Thank you!

Hi,
I don't see a p4_hlir in the submodules, so I am not sure what you are referring to. This repository assumes that p4-hlir has been installed, just like in this script: https://github.com/p4lang/p4factory/blob/master/install_deps.sh#L101
Finally, I notice that you are using the first version of the behavioral model here (bmv1), we are planning on deprecating it soon in favor of bmv2 (https://github.com/p4lang/behavioral-model). See here for more details: https://github.com/p4lang/p4factory#towards-a-better-behavioral-model-bmv2


WIP - Support for Solidus v1.2.2

This pull request adds support for Solidus v.1.2.2
Note: This won't be merged, it will remain in its own branch since we plan on maintianing backward compatibility with spree 3.0

Just wanted to post an update on this WIP branch. It seems Solidus introduced store credits which is breaking our spree extension.
When I try to checkout Komoju I'm receiving the following error message:

I've managed to trace the error to here
https://github.com/solidusio/solidus/blob/master/core/app/models/spree/order.rb#L654
I think this may be a bug in Solidus but need a bit more time this week to debug.

I tried with solidus 1.3.0 on sandbox.
It is working good konbini and credit card.
Is it ready for solidus?


Referrals - Invite friends (mini-app)

Create a service menu branding (role threefold.users.invited) name "Invite friends to ThreeFold" (or "Invite to ThreeFold" if the first is too long to fit on 2 lines).
Show 2 options:

Invite via SMS
Invite via E-mail

When clicking an option the SMS/E-mail app is opened and pre-filled with the following text.

Hi,
I'd like to invite you the ThreeFold community. You can download the Android or iOS app at https://rogerth.at/install?a=${rogethat.user.app_id}
To become a registered user in the app, you can use my invitation code ${rogethat.user.data.referral_code}.
Welcome on board!
${rogethat.user.data.name || rogethat.user.name}

Uploaded to the service


improve optimum-habana available check

Type of Change
clean up the code.
no api change.
Description
move the optimum habana available to environ.py, because  other available functions are in environ.py
Expected Behavior & Potential Risk
the expected behavior that triggered by this PR
How has this PR been tested?
how to reproduce the test (including hardware information)
Dependency Change?
any library dependency introduced or removed

code coverage decrease 0.132%, is it ok to merge? @XuehaoSun


Is there any way to include two sticky sidebar.

I have 2 sidebar so I want to allow both to be sticky.
=> Left menu bar
=> Right Cart

Left hand side menu bar overlap footer when I have collapsible menu category list.

I think it would be better if you made your both sidebars fixed, and add overflow-y: scroll to your main content, dont you think?
let me know if this could solve your problem.


[docs] add more docs for various utils in the typescript SDK

Description
This adds docs for a few missing use cases in the SDK
Test Plan
How did you test the new or updated feature?

If your changes are not user-facing and not a breaking change, you can skip the following section. Otherwise, please indicate what changed, and then add to the Release Notes section as highlighted during the release process.
Type of Change (Check all that apply)

[ ] protocol change
[ ] user-visible impact
[ ] breaking change for a client SDKs
[ ] breaking change for FNs (FN binary must upgrade)
[ ] breaking change for validators or node operators (must upgrade binaries)
[ ] breaking change for on-chain data layout
[ ] necessitate either a data wipe or data migration

Release notes

thanks @randall-Mysten for the review! great feedback, writing docs is definitely not one of my strengths.


I cannot access contact details from email after match

Describe the bug
A clear and concise description of what the bug is.
To Reproduce
Steps to reproduce the behavior:

click teams anzeigen

I guess we don't show any contact details here on purpose since we want the user to click the contact button in order to track if a match is interesting?
click contact button

I guess we don't show any contact details here on purpose since we want the user to click the contact button in order to track if a match is interesting?

Correct

click contact button

The link is access protected and can only be viewed by the user who is the recipient of the match. Most likely you were not logged-in as that user but the other one?

duplicate of #221


update eigen3-nnls

Fix _P namespace conflict for android/gcc.

Testing:

https://ci.appveyor.com/project/ingenue/hunter/build/1.0.2604
https://travis-ci.org/ingenue/hunter/builds/291192887?utm_source=github_status&utm_medium=notification

Uploading:

https://ci.appveyor.com/project/ingenue/hunter/build/1.0.2606
https://travis-ci.org/ingenue/hunter/builds/291203428?utm_source=github_status&utm_medium=notification

Released: https://github.com/ruslo/hunter/releases/tag/v0.19.131

Tests == 👍


[Bug Report] FAILED models/demos/resnet/demo/demo.py::test_demo_sample[20-models/demos/resnet/demo/images/-device_params0] - RuntimeError: TT_FATAL @ ../tt_metal/impl/allocator/allocator.cpp:123: num_shards.value() <= num_compute_banks

Describe the bug
I followed this paragraph to test resnet.
To Reproduce
Steps to reproduce the behavior:

Download tt-metal and build it based on the https://github.com/tenstorrent/tt-metal/blob/main/INSTALLING.md
Run export TT_METAL_ENV=dev && export TT_METAL_HOME=$PWD && export ARCH_NAME=grayskull && export PYTHONPATH=$TT_METAL_HOME
And run pytest --disable-warnings models/demos/resnet/demo/demo.py::test_demo_sample[20-models/demos/resnet/demo/images/-device_params0]

Expected behavior
This is my first to run it. I expected it runs fine.
Screenshots
The attachment contains the error messages I got.
Please complete the following environment information:

OS: Ubuntu 20.04
Built from

commit dee91cfa86b077ddc0ff2bdd941172048693ce93 (HEAD -> main, tag: v0.51.0-rc11, origin/main, origin/HEAD)
Author: Colman Glagovich<EMAIL_ADDRESS>Date:   Tue Jul 23 20:17:09 2024 -0400

    Update README.md with Llama3.1 numbers (#10630)

Additional context

Hardware: GraySkull e75
log.txt

@jaebaek , most demo models do not support e75 and are aiming higher end cards, including the resnet model.
It is a gap in our doc and I am working with the team now to address it.

Thanks. Let me close this issue.


Filter chain halted as :login_required rendered or redirected

When accessing web-interface with admin email / password, I'm facing error in rails.log
Also, latest version of Haml doesn't work for me. I needed to specify version 5.0.4 explicitly.
Started GET "/login?return_to=%2F" for xxx.xxx.xxx.xxx at 2018-11-29 07:42:11 +0000
Processing by SessionsController#new as HTML
  Parameters: {"return_to"=>"/"}
  Rendering sessions/new.html.haml within layouts/sub
  Rendered sessions/new.html.haml within layouts/sub (1.2ms)
Completed 200 OK in 3ms (Views: 2.8ms | ActiveRecord: 0.0ms)
Started GET "/" for xxx.xxx.xxx.xxx at 2018-11-29 07:42:12 +0000
Processing by OrganizationsController#index as HTML
Redirected to https://domain.com/login?return_to=%2F
Filter chain halted as :login_required rendered or redirected
Completed 302 Found in 1ms (ActiveRecord: 0.0ms)
Started GET "/login?return_to=%2F" for xxx.xxx.xxx.xxx at 2018-11-29 07:42:12 +0000
Processing by SessionsController#new as HTML
  Parameters: {"return_to"=>"/"}
  Rendering sessions/new.html.haml within layouts/sub
  Rendered sessions/new.html.haml within layouts/sub (1.2ms)
Completed 200 OK in 3ms (Views: 2.5ms | ActiveRecord: 0.0ms)
Started GET "/" for xxx.xxx.xxx.xxx at 2018-11-29 07:42:12 +0000
Processing by OrganizationsController#index as HTML
Redirected to https://domain.com/login?return_to=%2F
Filter chain halted as :login_required rendered or redirected
Completed 302 Found in 1ms (ActiveRecord: 0.0ms)
Started GET "/login?return_to=%2F" for xxx.xxx.xxx.xxx at 2018-11-29 07:42:12 +0000
Processing by SessionsController#new as HTML
  Parameters: {"return_to"=>"/"}
  Rendering sessions/new.html.haml within layouts/sub
  Rendered sessions/new.html.haml within layouts/sub (1.2ms)
Completed 200 OK in 3ms (Views: 2.4ms | ActiveRecord: 0.0ms)
Started POST "/login" for xxx.xxx.xxx.xxx at 2018-11-29 07:42:19 +0000
Processing by SessionsController#create as HTML
  Parameters: {"utf8"=>"✓", "authenticity_token"=>"XXXXX", "return_to"=>"/", "email_address"=>"email.com", "password"=>"[FILTERED]", "commit"=>"Login"}
Completed 500 Internal Server Error in 103ms (ActiveRecord: 0.4ms)
NoMethodError (undefined method `host=' for #<Authie::Session:0x00007fcc5415a3b0>):
app/controllers/application_controller.rb:127:in `login'
app/controllers/sessions_controller.rb:9:in `create'

Idk what happened, but fresh installation with ruby2.3 worked (used ruby 2.5 previously).

Thanks for letting us know, I think there are a few Ruby 2.5 related issues floating around.


Revert "Upgrade psycopg2 on Python 3"

Reverts DataDog/integrations-core#9590.
It is breaking the datadog-agent build because of the missing pg_config

I'm curious why this wasn't identified as an issue in CI?

@djmitche I think it was caught on CI according to https://github.com/DataDog/integrations-core/pull/9590#issuecomment-866979137, but maybe @ofek forgot to do the other change first or he thought it was already done


Cannot create action with a name that collides with a Python library

We created an action called "elasticsearch.py" (with the appropriate yaml). When we try to run it we get an ST2 exception. We are able to reproduce with another action called "test.py". Our hypothesis is that if the action name is the same as a Python library, it fails to run it. This happens after migrating to 1.4.0 from 0.13.2 (it works fine in 0.13.2).
Here's the stacktrace:
$ st2 run netflix_samples.elasticsearch app_region=us-east-1
.
id: 57211d9b7279e6326f6b025f
status: failed
parameters:
 app_region: us-east-1
result:
 exit_code: 1
 result: null
 stderr: "Traceback (most recent call last):
 File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2actions/runners/python_action_wrapper.py", line 164, in <module>
   obj.run()
 File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2actions/runners/python_action_wrapper.py", line 102, in run
   action = self._get_action_instance()
 File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2actions/runners/python_action_wrapper.py", line 116, in _get_action_instance
   actions_cls = action_loader.register_plugin(Action, self._file_path)
 File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2common/util/loader.py", line 163, in register_plugin
   (plugin_abs_file_path))
Exception: Found no classes in plugin file "/opt/stackstorm/packs/netflix_samples/actions/elasticsearch.py" matching requirements.
"
 stdout: ''

Same issue with  pack names
Example: can't have a pack named datadog if did "pip install datadog"

@jsjeannotte Can you share contents of test.py and test.yaml where you see this issue?

https://github.com/StackStorm/st2/blob/master/st2common/st2common/util/loader.py#L55
This is because of how we use some python module loading  hackery to import classes with actions in them. I can point you to the code but fixing it would require us to move to stevedore like entry_point approach which then makes the whole pack story horrendous IMO.

Suspect it might originate https://github.com/StackStorm/st2/blob/master/st2common/st2common/util/loader.py#L130.
We might need to pass in the pack_name as per https://docs.python.org/2/library/importlib.html#importlib.import_module although there is no guarantee of a package being present from the python perspective. This might be in @Kami wheelhouse.

Any updates on this issue?

@skarmark Since there are work arounds for this and really it seems like a trivial problem to me, can we wait on this?

We can wait on this. It is not super urgent but problematic nonetheless. We give freedom to our consumers to select their action and action file names and we have to catch these errors at runtime and ask them to change the action name which is not a very good user experience. Also, changing the action name which is deployed and possibly running is tricky.

I will look into it some time in the next couple of days and see if it's possible to do an easy fix.

After looking into the issue again more in-depth I don't think we should change this behavior - right now it behaves the same as any other code in Python.
In regular Python there are some ways around it (e.g. using from future import absolute_import), but in our case only possible work-around to get it to work would be to to manipulate PYTHONPATH for the Python action sub-process, but this would cause problems since it would mask other imports.
Another option would be to make action/ directory a package (this would require user to add init.py file to the directory), but I'm not a big fan of this approach since actions are not necessary Python only.
In any case, I will still play around with it a bit and see if we can come up with a compromise - perhaps make a package thing optional and if init.py is not present, it will behave as it does right now.

@skarmark #2870 should address it, but I still need to do more testing to make sure it doesn't break or negatively change the way things currently work.

@skarmark #2870 has been merged into master. If you feel adventurous, you could try new v1.7dev packages once they are built.

Thank you!!


⚠ SkipTheCommericals has degraded performance

In bea6cd8, SkipTheCommericals ($STC) experienced degraded performance:

HTTP code: 200
Response time: 1083 ms

Resolved: SkipTheCommericals performance has improved in 2529c42.


Support Python3 for discovery module

Hi.
I start supporting Python3.
First of all, discovery module is first. And I'll keep supporting another modules.
Thanks.

i think i'd prefer to use six rather than play games with imports. (this is the recommended approach from what i saw in the py3 porting guide?)

@craigcitro Oh I forgot the six. I'll commit using six.

@craigcitro I fixed using six.

I think that this pull request may no longer be necessary as of cf939a33f6b8 - please confirm and take action?


chore(queue): add queue as pkg

Description
This PR is a follow-up from a previous PR that introduced a queue for the administration event stream interface.
The backing queue was generic, so it's been pulled into a generic package, usnig generic types as well as exposing optional max size and metrics settings.
Checklist

[ ] Investigated and inspected CI test results
[ ] Unit test and regression tests added
[ ] Evaluated and added CHANGELOG entry if required
[ ] Determined and documented upgrade steps
[ ] Documented user facing changes (create PR based on openshift/openshift-docs and merge into rhacs-docs)

If any of these don't apply, please comment below.
Testing Performed
Here I tell how I validated my change

see added unit tests; as well as existing unit tests for administration events passing in CI.

Reminder for reviewers
In addition to reviewing code here, reviewers must also review testing and request further testing in case the
performed one does not seem sufficient. As a reviewer, you must not approve the change until you understand the
performed testing and you are satisfied with it.

Current dependencies on/for this PR:

master

PR #7884   👈

This comment was auto-generated by Graphite.


Removed HISTORY.rst file and changed dev status to Alpha

Trent said we should change the status of the BigchainDB Python Driver from pre-alpha to alpha, so I did that.
I also deleted the HISTORY.rst file and everything that referenced it. It's a nice idea, but we have better things to do than maintain that (something we weren't doing anyway). Git has the code history and if you really want a list of all bigchaindb_driver packages that PyPI has ever had, see:
https://pypi.python.org/simple/bigchaindb-driver/

Current coverage is 100% (diff: 100%)

Merging #135 into master will not change coverage

@@           master   #135   diff @@
====================================
  Files           7      7
  Lines         135    135
  Methods         0      0
  Messages        0      0
  Branches        0      0
====================================
  Hits          135    135
  Misses          0      0
  Partials        0      0

Powered by Codecov. Last update b6a4599...c94488b

I can recommend using travis pypi integration with Github releases as I did it with bigchaindb_common.

@ttmc The one commit addresses two very separate and different topics. I understand that they both may be very minor, but in general, it is good practice to have commits that address one thing.

Is this related to this PR in anyway?

Yes. You can log (breaking) changes in the notes of a release.

Yes. You can log (breaking) changes in the notes of a release.

Ok. But the first question that will need to be addressed before doing so is whether we do want to do automated releases for the driver. As of now, the answer to this question was no. We can revisit the question, but  this is a separate issue.
And for now there is a working boilerplate, which has been put into place, and I do not see reasons for changing it at this very moment. The focus should be on the development of the driver itself, not on its boilerplate, which is really solid, and based on best practices from the community.

I will close this PR and I'll make a new one where I only change the status to Alpha.


print-case :capitalize does not capitalize printed values

*print-case* :capitalize behaves the same as *print-case* :downcase in Corman CL.  But, it works fine in SBCL.
Corman Lisp 3.1  (Patch level 2) Example: Note that the "m" in the last "mike" is NOT properly capitalized.  The entire name is lowercase as if the :downcase keyword were used instead of :capitalize.
?(defvar *name* (read))
mike
*NAME*
?*name*
MIKE
?(setq *print-case* :capitalize)
:capitalize
?*name*
mike

SBCL 1.4.14 Example: Note that the "M" in the last "Mike" is properly capitalized while the rest of the name is properly lowercase.
* (defvar *name* (read))
mike
*NAME*
* *name*
MIKE
* (setq *print-case* :capitalize)
:Capitalize
* *name*
Mike

@ninejaguar
Does the attached build work as it should?
CormanLispBinaries.zip

It works!  Thank you for the quick fix!
(defvar *name* (read))
mike
*NAME*
MIKE
(setq *print-case* :capitalize)
:Capitalize
*NAME*
Mike

@ninejaguar You are welcome! You have provided a lot of meaningful feedback recently, thank you for your interest in Corman Lisp!


Update reference.md to fix typo and add a port for Nginx viewer

Update reference.md to fix typo and add a port for Nginx viewer

You are correct,  I will close this.  I tried to keep them separate but they got munged.  My bad.


[SPARK-14504][SQL] Enable Oracle docker tests

What changes were proposed in this pull request?
Enable Oracle docker tests
How was this patch tested?
Existing tests

Test build #55419 has started for PR 12270 at commit 9f65ab3.

This PR is dependent on PR #9893

Test build #55419 has finished for PR 12270 at commit 9f65ab3.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55419/
Test FAILed.

I just merged #9893 so this should be ready for a rebase.

Ping RE: rebase.

Test build #56018 has started for PR 12270 at commit f0790d5.

Test build #56018 has finished for PR 12270 at commit f0790d5.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/56018/
Test FAILed.

Merged build finished. Test FAILed.

Test build #56019 has started for PR 12270 at commit a78c9a7.

Test build #56019 has finished for PR 12270 at commit a78c9a7.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/56019/
Test FAILed.

Merged build finished. Test FAILed.

Test build #56021 has started for PR 12270 at commit bb6658a.

Test build #56021 has finished for PR 12270 at commit bb6658a.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/56021/
Test FAILed.

Test build #56024 has started for PR 12270 at commit 71b99d8.

Test build #56024 has finished for PR 12270 at commit 71b99d8.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/56024/
Test FAILed.

Merged build finished. Test FAILed.

Test build #56030 has started for PR 12270 at commit 1490ca0.

Test build #56030 has finished for PR 12270 at commit 1490ca0.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/56030/
Test FAILed.

Test build #56031 has started for PR 12270 at commit 5b17a11.

Test build #56031 has finished for PR 12270 at commit 5b17a11.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/56031/
Test FAILed.

Test build #56042 has started for PR 12270 at commit f281b43.

Test build #56042 has finished for PR 12270 at commit f281b43.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/56042/
Test FAILed.

Merged build finished. Test FAILed.

Test build #56062 has started for PR 12270 at commit a5ccec6.

Test build #56062 has finished for PR 12270 at commit a5ccec6.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/56062/
Test FAILed.

Jenkins, retest this

Jenkins, retest this please.

Test build #56087 has started for PR 12270 at commit 9f92dc1.

Test build #56110 has started for PR 12270 at commit 6b700e4.

Test build #56087 has finished for PR 12270 at commit 9f92dc1.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/56087/
Test FAILed.

Test build #56110 has finished for PR 12270 at commit 6b700e4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/56110/
Test PASSed.

Merged build finished. Test PASSed.

I'm a bit skeptical of some of the build hacks needed for this but out of pragmatism I'm going to merge this now and will just remove these tests later if this turns out to be unreliable.

These tests are too flaky. I've already seen a few failures.
We should disable them from the normal tests and maybe occasionally running them (via some trigger or just run it once before the release).

I just failed to build Spark locally once due to the docker oracle test.

@rxin, just trying to understand, is the oracle test the only one failing ? Or you are suggesting we move the whole docker based tests to a separate profile that would only run on Jenkins ?

I don't even think they should run on pull requests. Tests that require extensive external setup (or downloading things) in general are flaky.

@rxin Let me move them to a specific docker profile. But I would still run them on Jenkins, as the infrastructure is already setup there.

They have been flaky and causing other pull requests to fail. That's why we shouldn't run them on Jenkins.

We can run them via some trigger phrase though.

Ok, I will work with @JoshRosen on the trigger part.


Diagnose filtering issue

The query filtering is not working as expected. For example, issuing the following GraphQL query will not correctly filter the list:
  listStrains(input: { limit: 10, filter: "species==='Dictyostelium purpureum'" }) {
    strains {
      id
      label
      species
    }
  }

The result just provides a list of all strains like so:
{
    "id": "DBS0351367",
   "label": "myoB-/[act15]:myoB(S332A)",
   "species": "Dictyostelium discoideum"
},

The issue appears to be when a filter contains a space. For example, changing the filter to filter: "species~purpureum" returns the correct set of results.
It's possible that the filter statement is not getting parsed and/or generated properly in arangomanager. However, need to follow the chain of logic to properly diagnose.

The query listed above for Dictyostelium purpureum now works since arangomanager has been updated to add quotes for specific string queries.


Format package.json files with Prettier

Finishing the work started here: https://github.com/Polymer/lit-html/pull/1351
Now use Prettier plugin (prettier-plugin-package) to auto-format the package.json files: all the fields in the same order.
This had been temporarily removed due to changing the order of the files field.

Reported here: https://github.com/shellscape/prettier-plugin-package/issues/12
Fixed here: https://github.com/shellscape/prettier-plugin-package/pull/17
And now it's released in the latest version.

@aomarks please check it when you can!

Friendly ping to @aomarks!
Now we also have it at webcomponents/polyfills repository: package.json#L48

Friendly ping to @aomarks!
Now we also have it at webcomponents/polyfills repository: package.json#L48

@aomarks maybe you can review this?


Eschnett/floattypes

Coverage remained the same at 79.521% when pulling 3f14bba482fe61b9661d93fde8f6968be2c3cc03 on eschnett/floattypes into 4d220db14bd11152a5b2431701dc85f9fd45beb2 on master.


CSV not breaking lines

Include the version of json2csv used.
V 5.0.6

Include your node version/browser vendor and version.
Node 14.17

Include a sample dataset that we can test against.
Running it on a list of items shaped like:

{
    item_url: 'url',
    lead_source: 'site',
   item_notes: 'somewhere',
   company_name: 'some company',
   business_phone: '555-555-5555',
   owner_id: '123'
  },
  ...

Tried to generate the file in many ways, for example a very basic attempt:

  const fields = [
    "item_url",
    "lead_source",
    "item_notes",
    "company_name",
    "business_phone",
    "owner_id",
  ]
  const opts = { fields }

  const csv = parse(res, opts)

  await saveDataToFile(csv, `short-url-list-${new Date().toISOString()}`, true)

Include your output/error.
The result ends up being a single line CSV file like here:

The new line tag is being ignore.

That's seems to be an issue with whatever you are using to open the CSV.
I just run you example and it produced the new lines correctly.
Check that you are using the right line ending (linux vs windows).

That's seems to be an issue with whatever you are using to open the CSV.
I just run you example and it produced the new lines correctly. Check that you are using the right line ending (linux vs windows).

Got thanks I will try, for the past few times we used it I basically exported a JSON as those were properly formatted for some reason, I will take a look at our CSV reader solution next.


Block spacing not working in 0.12.0 Testing

Description
MWE:
#set block(spacing: 10pt)

#let f(a) = {
    set block(spacing: 1pt)
    a
}

#lorem(10)

#lorem(10)

#lorem(10)

#f[
    #lorem(10)

    #lorem(10)

    #lorem(10)
]

0.11.1:

0.12.0 Testing:

Reproduction URL
No response
Operating system
No response
Typst version

[X] I am using the latest version of Typst

This was an intentional breaking change. See https://github.com/typst/typst/pull/4390.

I see. Maybe breaking changes deserve their own section in the Changelog for better visibility.


Enum Array List Parceling

Added the ability for the ArrayListParceler to parcel Arrays with Enums

@ndtaylor what formatter are you using for this project? Getting a check style error.

@ndtaylor what formatter are you using for this project? Getting a check style error.

Nevermind I found your code style xml file. I'll patch this PR asap. On another note the tests for handling array lists of enums seems to be working but prior to this change I saw exceptions thrown by the annotation processor when trying to handle array lists of enums. Does there need to be an update around that logic? And if so, where do I find that code?

I'll take a look this evening and let you know.
On Mon, Mar 28, 2016, 10:07 Kenneth Nickles<EMAIL_ADDRESS>wrote:

@ndtaylor https://github.com/ndtaylor what formatter are you using for
this project? Getting a check style error.
Nevermind I found your code style xml file. I'll patch this PR asap. On
another note the tests for handling array lists of enums seems to be
working but prior to this change I saw exceptions thrown by the annotation
processor when trying to handle array lists of enums. Does there need to be
an update around that logic? And if so, where do I find that code?
—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/Workday/postman/pull/24#issuecomment-202489641

Postman 1.2 added relevant functionality


A method to bypass Doze mode?

I want to use this wondeful library for an Alarm app. In Android O emulator it runs perfectly, but in physical Android M there is always 4-25min delay. Probably because Doze mode. Including app in whitelist didn't work either. Thanks.
new JobRequest.Builder(tag)
.setExact(millisecondsFromNow)
.setExtras(extras)
.build()
.schedule();

That snippet should work without any problems. Can you provide a full sample where it's broken?

Closing because of inactivity.


Support Response return type serialization in APIServices

I use Response wrapped response structure everywhere in my services (for error handling) so it looks like:
interface Service {
  @GET("string")
  suspend fun getAll(): Response<List<String>>

  @GET("string/{id}")
  suspend fun getSingle(@Path("id") id: Long): Response<String>
}

And then in the ktor-retrofit service implementation, I then return a
return Response.success(myResponseObj)
But for some reason, my responses are not transmitted if implemented this way, here's a snippet of the response when hitting a ktor-retrofit implemented endpoint (logged via HttpLoggingInterceptor):
Vary: Origin
Content-Type: application/json; charset=UTF-8
Connection: keep-alive
transfer-encoding: chunked

{}
<-- END HTTP (2-byte body)

Notice the empty {}
The service is installed as:
  routing {
        route("/") {
            authenticate(AuthLabel) {
                retrofitService(service = MyService)
            }

So I've got this coded up and working but I'm a little hesitant to support this. retrofit2.Response on the server side seems like a leak of dependencies. Right now, if the Retrofit interface is in a separate, common module, the Retrofit dependency only needs to be present at runtime. Using Response for the interface implementation forces Retrofit to be present at compile time as well.
Is Response being used in the interface a requirement from the client side? Is there a reason to avoid HttpException? I'm just trying to better understand your use case before releasing support for this.

Is Response being used in the interface a requirement from the client side?

Yes, actually.
Here's an example method of my user API service interface:
    @GET(V1_USER)
    suspend fun getUser(): Response<UserResponse>

So, in order to use the latest coroutine support with retrofit (along with wrapping API calls in live data on Android for good measure) using Response is pretty much the only solution. Mere HttpExceptions don't allow as much detailed handling of errors (in situations where the errors cannot just be represented by HTTP status codes). That leaves RxJava Observable<T> or raw data type handling, both of which are (IMO) worse solutions than good old Response<T>
Hence, if I can implement the same user api service, as is, on the backend using ktor-retrofit, that pretty much saves me an entire file of duplicated API endpoint paths and routing to specific methods (the usual ktor way, except without ktor-retrofit's explicit type safety, again a worse solution)
I'm wondering how common of a situation it would be where retrofit would be used at runtime but not compile time. If ktor-retrofit service() makes use of a Service in a common module, isn't it already present at compile time? (Unless someone's using reflection to load Retrofit services, which is a really bad idea for so many reasons)
Worst case, the Response handling could be released as an add-on dependency perhaps? (Kinda like how you can add a converter factory to retrofit)
Also, retrofit-2.6.2.jar is 124 Kb (before minification), not really a big deal when we're talking about server deployment.

Okay, thanks for the explanation. I'll hopefully have a new release out soon.

Fixed by 5d85a44e7b0cebec2aa7ad678dd0e7cd5bb5bf75 and available in latest 0.1.2 version.


dedupe command doesn't update product grade

the dedupe command is running the deduplication on findings, but the product grade is not updated accordingly.

fixed in dev / 1.15.0


[gardening] Flaky standalone/io/http_response_deadline_test/

There are new test failures on [vm] Clean up New() methods of Dart classes....Bump linter to 655de5d0dc30cdd1df9ce4cd16bd7489106d987f.
The tests
standalone/io/http_response_deadline_test/2 RuntimeError (expected Pass)

are failing on configurations
vm-linux-release-arm64

Meta flake history: https://dart-ci.web.app/#showLatestFailures=false&test=standalone/io/http_response_deadline_test/2&configurations=vm-linux-release-arm64
However, this seems to be a new failure:
/=========================================================================================\
| standalone/io/http_response_deadline_test/1 broke (Pass -> RuntimeError, expected Pass) |
\=========================================================================================/

--- Command "vm" (took 22.000826s):
DART_CONFIGURATION=ReleaseXARM64 out/ReleaseXARM64/dart --short_socket_read --sound-null-safety -Dtest_runner.configuration=vm-linux-release-arm64 --ignore-unrecognized-flags --packages=/b/s/w/ir/.dart_tool/package_config.json /b/s/w/ir/tests/standalone/io/http_response_deadline_test.dart

exit code:
255

stderr:
Unhandled exception:
Expect.equals(at index 0: Expected <stuff>, Found: <missing or invalid authentication code...>) fails.
#0      Expect._fail (package:expect/expect.dart:696:5)
#1      Expect.equals (package:expect/expect.dart:115:9)
#2      testDeadlineAndDetach.<anonymous closure>.<anonymous closure>.<anonymous closure> (file:///b/s/w/ir/tests/standalone/io/http_response_deadline_test.dart:84:18)
<asynchronous suspension>
#3      Future.wait.<anonymous closure> (dart:async/future.dart:523:21)
<asynchronous suspension>
#4      testDeadlineAndDetach.<anonymous closure>.<anonymous closure> (file:///b/s/w/ir/tests/standalone/io/http_response_deadline_test.dart:88:31)
<asynchronous suspension>

--- Re-run this test:
python3 tools/test.py -n vm-linux-release-arm64 standalone/io/http_response_deadline_test/1

https://logs.chromium.org/logs/dart/buildbucket/cr-buildbucket/8775967008881223121/+/u/test_results/ignored_flaky_test_failure_logs
The flaky failures look like:
/=========================================================================================\
| standalone/io/http_response_deadline_test/3 broke (Pass -> RuntimeError, expected Pass) |
\=========================================================================================/

--- Command "vm" (took 19.000220s):
DART_CONFIGURATION=ReleaseXARM64 out/ReleaseXARM64/dart --short_socket_read --short_socket_write --sound-null-safety -Dtest_runner.configuration=vm-linux-release-arm64 --ignore-unrecognized-flags --packages=/b/s/w/ir/.dart_tool/package_config.json /b/s/w/ir/tests/standalone/io/http_response_deadline_test.dart

exit code:
255

stderr:
Unhandled exception:
Expect.fail('Expected error')
#0      Expect._fail (package:expect/expect.dart:696:5)
#1      Expect.fail (package:expect/expect.dart:280:5)
#2      testExceedDeadline.<anonymous closure>.<anonymous closure> (file:///b/s/w/ir/tests/standalone/io/http_response_deadline_test.dart:50:16)
<asynchronous suspension>
#3      Future.wait.<anonymous closure> (dart:async/future.dart:523:21)
<asynchronous suspension>
#4      testExceedDeadline.<anonymous closure>.<anonymous closure> (file:///b/s/w/ir/tests/standalone/io/http_response_deadline_test.dart:55:31)
<asynchronous suspension>

--- Re-run this test:
python3 tools/test.py -n vm-linux-release-arm64 standalone/io/http_response_deadline_test/3

https://logs.chromium.org/logs/dart/buildbucket/cr-buildbucket/8776194953081260753/+/u/test_results/ignored_flaky_test_failure_logs

@brianquinlan you're in charge of dart:io now right? I don't think I've ever seen such error before in a test locally setting up an HTTP connection.