ci: Add ci to the project

Prerequisites checklist

[ ] I have read the contributing guidelines.

What is the purpose of this pull request?
Set up CI Workflow for Linting and Formatting.

What changes did you make? (Give an overview)
Related Issues
Is there anything you'd like reviewers to focus on?

It looks like this triggered some linting errors. Check the "Files changed" tab to see them.
I checked, and there were two warnings

React Hook useCallback received a function whose dependencies are unknown. Pass an inline function instead
https://github.com/eslint/code-explorer/blob/c1e0d56c0edf859d2eac08a7fa346a9a9bda7668/src/components/editor.tsx#L53-L57
In the editor.tsx file, the onChange function we pass to useCallback receives a value as a prop to the component, which makes it unknown. Therefore, we can't use it as an inline function, so let's disregard this warning.

React Hook useEffect has a missing dependency: 'fetchCodePath'. Either include it or remove the dependency array
https://github.com/eslint/code-explorer/blob/c1e0d56c0edf859d2eac08a7fa346a9a9bda7668/src/components/path/index.tsx#L59-L61
In the useEffect code block, we can't include fetchCodePath in the dependency array because we only want to call it during the initial render, not every time its dependencies change. Therefore, this change is unnecessary.

Therefore let's ignore both warnings.


python: automatically generate facets and core structures from JsonSchema

Since beginning of OpenLineage, facets and core structures are defined by JsonSchema. Java client is generated using that data. Python client is still not. This is a source of some bugs and adds more work for spec authors - it should be automated.
The difficulty with that task is that Java has very useful JavaPoet library that takes care of serializing AST to working Java code.
Python does not have a library like that.
Since we're already parsing JsonSchema using Java, it makes sense to create skeleton library providing similar API, but for Python - and reusing JsonSchema generation from existing Java generator.

@JDarDagran Should this be closed now that https://github.com/OpenLineage/OpenLineage/pull/2520 is merged?


The spec passes locally. 🤔

The spec passes locally. 🤔
Originally posted by @rymai in https://github.com/danger/danger/pull/1333#issuecomment-951938397`` hello world

Its is not clear what this issue relates to, but it doesn't appear to be the OpenAPI Specification itself.


Installation of SpeedSeq by git error!

I try to install SpeedSeq by "git clone" and make, but some of the folders in "./speedseq/src" are empty, such as parallel, samblaster etc.
I wonder if some dependant tools have been modified, and would you please check all these tools? @cc2qe

I met the same problem. Please refer to this page "https://github.com/hall-lab/speedseq/issues/42".

Duplicate of https://github.com/hall-lab/speedseq/issues/42


Very bad performance - need help troubleshooting

We are using the Graph API to maintain a Graph of phone handsets calling each other and their voice quality metrics as the calls traverse the network.  Our data model is about 16 vertex classes and 32 edge types.

@aag-tonesoft could we have more information about the bad performance? It's in query, insertion, update, delete, everything? Could you show the most slow operations?

Inserts/updates are taking longer than expected.  We have a multi-threaded system collecting data, but have them filtered down to one thread because I was getting OConcurrentModification exceptions while added vertices and edges to the graph.  The delay in creating the edges and vertices is bearing itself out as a backlog of work and we quickly hit some tipping point where we can no longer keep up.  The basic jist is that we have a Call Vertex representing a call between two devices. We create edges between the call and a "bucket" of quality of the call.    There are generally 4 types of metrics we are collecting for a call, so we lookup or create a Call Vertex. We then look at the quality of the call, for metric A, lookup the bucket Vertex for metric A and create an edge from the call to the metric A bucket Vertex.  We repeat for metrics B,C, and D.  That represents a relationship of a call to the quality buckets, but our design needs to also know when the call occurred and for that point in time, what were the quality metrics. So, on a lot of edges we must maintain the time that the call occurred, the edges between the Call and metric bucket vertices. Additionally, there is a time tree of vertices to help us narrow down the calls that occurred during some time search.  The vertices in this tree looks like this Year -> Month -> Day -> Hour -> 5minIncrements -> Call -> [4 to 8 buckets].
The edges between [5minIncrement -> Call] and [Call -> Bucket] have a time stamps that I search on, and therefor there are quite possibly MANY edges between different 5MinIncrement Vertices pointing to a single Call Vertex AND MANY edges out of Call that single Vertex to the various buckets.  I think this insertion/update of vertices and edges is slow.
What tools are available to monitor slow operations. I see that profiling is turned on be default in the db, but I don't know how to use it. When i tried to use the console to 'profile dump' it had some error.
How can I give you a backup of our DB for you to look through.  I can give you a Gremline Pipeline in java to represent some of the queries we do.  in psuedocode, we're generally doing something like this: I call the Hour vertex TimeSegment and the 5minIncrement TimeSlices.  Let's assume I want a full hour of data, g.V(the hour I care about).out('TimeSegmentTimeSlice').out("TimeSliceCall").has(some indexed property segmenting the data for a customer).groupBy{call}{call.outE('CallQosBucket').has(time matching the hour that I want)}.cap

@lvca When I update a property on the edge edge between Call and Bucket vertices, according to the documentation, the optimistic locking updates the version number of the Call and Bucket vertices, does the update to those vertices cascade to updating version numbers of other vertices it's connected to?
I'm too late in the dev cycle to implement the update/retry logic suggested to overcome the OConcurrentModificationException that might be thrown when updating vertices so I've added some code to batch updates through a single thread to work as fast as possible.  While that thread is working as fast as possible, the updates are taking 26-45 seconds and we're generating data MUCH MUCH faster than that. It's a losing battle. I need real help. I may have to report to the VP that this technology is not going to scale.

The version number is not updated in cascade, but involve only the records that are part of the change. I don't know very much about your use case, your database and your code. Since you already ended the development we're going to try to optimize your bottlenecks without breaking too much your code.
I have a few questions: How do you update edges? Are you doing a batch of updates in one transaction? Or you rather have one tx per update? Why you update edges instead of creating them with the final value? The relevant code would be also useful. You can send private things to support at orientdb.com by specifying your client id. Thanks.

I'll work on extracting some code for my slow updates. Is it possible to have a GoToMeeting to explain our use case and discuss optimizations?
In all honesty, I have a better design in my head as to a replacement data model but for now, I cannot eat the cost of the refactoring.
We have vertices that represent data between two devices, Calls, handset A calls handset B. Often, we get traceroute information that the call traversed.  For this Call, we get 4 different metrics that i care to record.  Normally, when we get the call, I search in the Graph using gremlin for the vertex that matches the call. If I find it, I'll be adding edges to it. If it doesn't exist, I'll create it. It had indexed properties to help me find it and it has edges from a time tree to help me narrow down the search. For each of the 4 metrics, the value is normalized to some bucket, some finite range if values that virtually mean the same thing. Maybe the possible range is from 0-500, so I normalize the values into increments of 50.  So, I'll find a vertex that matches this normalized value and create an edge between it and the call. I'll do this same thing for the remaining there metrics. Additionally the call may have traversed some network hops to get from headset A to Handset B.  A -> hop1 -> hop2 -> B.  I call each pair of neighbors A::hop1, hop1::hop2, hop2::B NetworkSegments.  I search for our create vertices to represent the NetworkSegmenrs. The same metric vertices I attached to the Call vertex will be attached to each NetworkSegment vertex.  The edges (named CallDataBuckets) between Call and Metric or NetworkSegment and the Metric represent a 5 minute aggregation of all the traffic that passed through the devices. When creating or updating the CallDataBucket edges, I have some properties on it that are being updated, the count, metrics min, max, total, and average.
Right now, for each transaction I try to collect ( based on system load ) and group a bunch of calls together in each transaction.
Does this paint a picture of the data model? I can upload a backup of a sample db, please give instructions of how to send it.

Why closed?

I must have done something incorrect to close it, the issue is definitely not closed, it's killing me right now.

@aag-tonesoft please could you send the backup (not export) to support at orientdb.com? Furthermore we'd need the most slower queries to understand how to optimize them. Usually if you use Gremlin, the first step is using SQL: most of the times is much faster.

I sent an email to support with an explaination of our current problem, a stack trace and a java flight recording to demonstrate blocked threads, can you associate that email with this thread.  Any insight into the problem will help.

@aag-tonesoft I see that you sent an email a while back to support. Were you assisted? Does this problem still exist? Please attach any additional information to this ticket if more support is needed.

I got a response by nothing that truly solved my problem. Performance is not what I would expect, it degrades severely after enough data is inserted to the db. Our app is write heavy and after a few million edges it takes multiple seconds to write a string of graph nodes.
Amit
On Fri, May 20, 2016 at 1:17 PM -0700, "jillray7" <notifications<EMAIL_ADDRESS>wrote:
@aag-tonesofthttps://github.com/aag-tonesoft I see that you sent an email a while back to support. Were you assisted? Does this problem still exist? Please attach any additional information to this ticket if more support is needed.

You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHubhttps://github.com/orientechnologies/orientdb/issues/5250#issuecomment-220708115

@aag-tonesoft Can I try to help on this? What's the piece of code that you use to write the edge? You could also close this and open a private issue if it's the case.

closing issue, we're reworking our approach.


Next release is version 3.0.1

Preparation work for maintenance release.

Looks pretty good, I'll see if I can test it on the weekend. The situation with the documentation is a bit strange. It looks like the original pdf was generated, but I don't know from where. The source seems to be missing from this repo. Did you convert to odt with libreoffice or do is that the source document?

The .sxw file is the original source, which I converted to .odt.
Form there I exported as .pdf, which is almost identical to the original PDF.
In the future I would want to convert this in a better format for writing documentation, for example asciidoc.
SXW was the former default format of Star Office/Open Office back then.

Here is the actual diff without the first commit:
https://github.com/pspdev/psplinkusb/compare/deb730d...b8ffe81


[Bug] ZFS mirroring undetected, both drive show 0%

Description of the bug
Hello, for some reason my ZFS pool (composed of 2 drives) is not detected at all, and the interface shows the 2 raid drives as empty. I would expect to see 1 drive instead, showing the total used space, along with something like "zfs pool" or "raid" written.

On the picture above, the first drive is where the os is installed : there is nothing wrong with this one.
Is there a mistake in my config or something I have overlooked ? More info below:
docker exec -t CONTAINER df -kPT outputs:
Filesystem           Type       1024-blocks    Used Available Capacity Mounted on
overlay              overlay    238169856  16634912 209363784   7% /
tmpfs                tmpfs          65536         0     65536   0% /dev
shm                  tmpfs          65536         0     65536   0% /dev/shm
/dev/sda2            ext4       238169856  16634912 209363784   7% /mnt/host
udev                 devtmpfs     8032108         0   8032108   0% /mnt/host/dev
tmpfs                tmpfs        8090464         0   8090464   0% /mnt/host/dev/shm
tmpfs                tmpfs        1618096      2004   1616092   0% /mnt/host/run
tmpfs                tmpfs           5120         0      5120   0% /mnt/host/run/lock
tmpfs                tmpfs        1618096      2004   1616092   0% /mnt/host/run/snapd/ns
tmpfs                tmpfs        1618092         4   1618088   0% /mnt/host/run/user/1000
/dev/sda1            vfat         1098628      6216   1092412   1% /mnt/host/boot/efi
pool_0               zfs<PHONE_NUMBER><PHONE_NUMBER>560   0% /mnt/host/mnt/pool_0
pool_0/media         zfs<PHONE_NUMBER> 503720704<PHONE_NUMBER>  29% /mnt/host/mnt/pool_0/media
pool_0/cloud         zfs<PHONE_NUMBER> 138841728<PHONE_NUMBER>  10% /mnt/host/mnt/pool_0/cloud
/dev/loop0           squashfs       56960     56960         0 100% /mnt/host/snap/core18/2721
/dev/loop1           squashfs       64896     64896         0 100% /mnt/host/snap/core20/1822
/dev/loop2           squashfs       64896     64896         0 100% /mnt/host/snap/core20/1852
/dev/loop3           squashfs      114688    114688         0 100% /mnt/host/snap/lxd/24322
/dev/loop4           squashfs       51072     51072         0 100% /mnt/host/snap/snapd/18357
/dev/loop5           squashfs       54528     54528         0 100% /mnt/host/snap/snapd/18933
overlay              overlay    238169856  16634908 209363788   7% /mnt/host/var/lib/docker/overlay2/79b9202a99396f0bfabddeec5805ec09b9a150c5bd8d63e21878cfe3e96e54e9/merged
overlay              overlay    238169856  16634908 209363788   7% /mnt/host/var/lib/docker/overlay2/d7696b0c7d11578c245e197c7061abde1dde04553a61576e29db213f08ebfc91/merged
overlay              overlay    238169856  16634908 209363788   7% /mnt/host/var/lib/docker/overlay2/c8b7ec3b45f441b77748d9a1f1c460e61bd831cba967990eb9a6a7efc6bf32da/merged
overlay              overlay    238169856  16634908 209363788   7% /mnt/host/var/lib/docker/overlay2/c7e4e21bf4dc9bf9e86599514afaa1b796c355b93a8573c63988634d0d3df8ab/merged
overlay              overlay    238169856  16634908 209363788   7% /mnt/host/var/lib/docker/overlay2/c7e4e21bf4dc9bf9e86599514afaa1b796c355b93a8573c63988634d0d3df8ab/merged
tmpfs                tmpfs          65536         0     65536   0% /mnt/host/var/lib/docker/overlay2/c7e4e21bf4dc9bf9e86599514afaa1b796c355b93a8573c63988634d0d3df8ab/merged/dev
shm                  tmpfs          65536         0     65536   0% /mnt/host/var/lib/docker/overlay2/c7e4e21bf4dc9bf9e86599514afaa1b796c355b93a8573c63988634d0d3df8ab/merged/dev/shm
/dev/sda2            ext4       238169856  16634908 209363788   7% /etc/resolv.conf
/dev/sda2            ext4       238169856  16634908 209363788   7% /etc/hostname
/dev/sda2            ext4       238169856  16634908 209363788   7% /etc/hosts

Pool_0 is reported as empty here, but I have read somewhere that it was a limit of the df command and is expected.
Running the same command on the host gives:
Filesystem     Type  1024-blocks      Used  Available Capacity Mounted on
tmpfs          tmpfs     1618096      2000    1616096       1% /run
/dev/sda2      ext4    238169856  16634932  209363764       8% /
tmpfs          tmpfs     8090464         0    8090464       0% /dev/shm
tmpfs          tmpfs        5120         0       5120       0% /run/lock
/dev/sda1      vfat      1098628      6216    1092412       1% /boot/efi
pool_0         zfs<PHONE_NUMBER><PHONE_NUMBER>560       1% /mnt/pool_0
pool_0/media   zfs<PHONE_NUMBER> 503720704<PHONE_NUMBER>      29% /mnt/pool_0/media
pool_0/cloud   zfs<PHONE_NUMBER> 138841728<PHONE_NUMBER>      11% /mnt/pool_0/cloud
tmpfs          tmpfs     1618092         4    1618088       1% /run/user/1000

And running zfs list on the host gives:
NAME           USED  AVAIL     REFER  MOUNTPOINT
pool_0         613G  1.16T       96K  /mnt/pool_0
pool_0/cloud   132G  1.16T      132G  /mnt/pool_0/cloud
pool_0/media   480G  1.16T      480G  /mnt/pool_0/media

My config is :
  dash:
    container_name: dash
    image: mauricenino/dashdot:latest
    restart: unless-stopped
    privileged: true
    # ports:
    #   - '${WEBUI_PORT_DASH:?err}:8080'
    volumes:
      - /:/mnt/host:ro
    environment:
      - DASHDOT_PORT=80
      - DASHDOT_ACCEPT_OOKLA_EULA=true
      - PUID=${PUID:?err}
      - PGID=${PGID:?err}
      - TZ=${TIMEZONE:?err}
    networks:
      - media-network

How to reproduce
Create a ZFS pool (using the zpool package) and launch dashdot
Relevant log output

Output:
const disks =  [
  {
    device: '/dev/sda',
    type: 'SSD',
    name: 'CT250MX500SSD1  ',
    vendor: 'ATA',
    size:<PHONE_NUMBER>16,
    bytesPerSector: null,
    totalCylinders: null,
    totalHeads: null,
    totalSectors: null,
    totalTracks: null,
    tracksPerCylinder: null,
    sectorsPerTrack: null,
    firmwareRevision: '045',
    serialNum: '',
    interfaceType: 'SATA',
    smartStatus: 'unknown',
    temperature: null
  },
  {
    device: '/dev/sdb',
    type: 'HD',
    name: 'WDC WD20EARS-00M',
    vendor: 'Western Digital',
    size:<PHONE_NUMBER>016,
    bytesPerSector: null,
    totalCylinders: null,
    totalHeads: null,
    totalSectors: null,
    totalTracks: null,
    tracksPerCylinder: null,
    sectorsPerTrack: null,
    firmwareRevision: 'AB51',
    serialNum: '',
    interfaceType: 'SATA',
    smartStatus: 'unknown',
    temperature: null
  },
  {
    device: '/dev/sdc',
    type: 'HD',
    name: 'WDC WD20EZRX-00D',
    vendor: 'Western Digital',
    size:<PHONE_NUMBER>016,
    bytesPerSector: null,
    totalCylinders: null,
    totalHeads: null,
    totalSectors: null,
    totalTracks: null,
    tracksPerCylinder: null,
    sectorsPerTrack: null,
    firmwareRevision: '0A80',
    serialNum: '',
    interfaceType: 'SATA',
    smartStatus: 'unknown',
    temperature: null
  }
]
const sizes =  [
  {
    fs: 'overlay',
    type: 'overlay',
    size:<PHONE_NUMBER>44,
    used:<PHONE_NUMBER>0,
    available:<PHONE_NUMBER>04,
    use: 7.36,
    mount: '/',
    rw: false
  },
  {
    fs: '/dev/sda2',
    type: 'ext4',
    size:<PHONE_NUMBER>44,
    used:<PHONE_NUMBER>0,
    available:<PHONE_NUMBER>04,
    use: 7.36,
    mount: '/mnt/host',
    rw: true
  },
  {
    fs: '/dev/sda1',
    type: 'vfat',
    size:<PHONE_NUMBER>,
    used: 6365184,
    available:<PHONE_NUMBER>,
    use: 0.57,
    mount: '/mnt/host/boot/efi',
    rw: true
  },
  {
    fs: 'pool_0',
    type: 'zfs',
    size:<PHONE_NUMBER>512,
    used: 131072,
    available:<PHONE_NUMBER>440,
    use: 0,
    mount: '/mnt/host/mnt/pool_0',
    rw: false
  },
  {
    fs: 'pool_0/media',
    type: 'zfs',
    size:<PHONE_NUMBER>336,
    used:<PHONE_NUMBER>96,
    available:<PHONE_NUMBER>440,
    use: 28.83,
    mount: '/mnt/host/mnt/pool_0/media',
    rw: false
  },
  {
    fs: 'pool_0/cloud',
    type: 'zfs',
    size:<PHONE_NUMBER>912,
    used:<PHONE_NUMBER>72,
    available:<PHONE_NUMBER>440,
    use: 10.05,
    mount: '/mnt/host/mnt/pool_0/cloud',
    rw: false
  },
  {
    fs: '/dev/loop0',
    type: 'squashfs',
    size: 58327040,
    used: 58327040,
    available: 0,
    use: 100,
    mount: '/mnt/host/snap/core18/2721',
    rw: false
  },
  {
    fs: '/dev/loop1',
    type: 'squashfs',
    size: 66453504,
    used: 66453504,
    available: 0,
    use: 100,
    mount: '/mnt/host/snap/core20/1822',
    rw: false
  },
  {
    fs: '/dev/loop2',
    type: 'squashfs',
    size: 66453504,
    used: 66453504,
    available: 0,
    use: 100,
    mount: '/mnt/host/snap/core20/1852',
    rw: false
  },
  {
    fs: '/dev/loop3',
    type: 'squashfs',
    size: 117440512,
    used: 117440512,
    available: 0,
    use: 100,
    mount: '/mnt/host/snap/lxd/24322',
    rw: false
  },
  {
    fs: '/dev/loop4',
    type: 'squashfs',
    size: 52297728,
    used: 52297728,
    available: 0,
    use: 100,
    mount: '/mnt/host/snap/snapd/18357',
    rw: false
  },
  {
    fs: '/dev/loop5',
    type: 'squashfs',
    size: 55836672,
    used: 55836672,
    available: 0,
    use: 100,
    mount: '/mnt/host/snap/snapd/18933',
    rw: false
  }
]
const blocks =  [
  {
    name: 'sda',
    type: 'disk',
    fsType: '',
    mount: '',
    size:<PHONE_NUMBER>16,
    physical: 'SSD',
    uuid: '',
    label: '',
    model: 'CT250MX500SSD1',
    serial: '',
    removable: false,
    protocol: 'sata',
    group: '',
    device: '/dev/sda'
  },
  {
    name: 'sdb',
    type: 'disk',
    fsType: '',
    mount: '',
    size:<PHONE_NUMBER>016,
    physical: 'HDD',
    uuid: '',
    label: '',
    model: 'WDC WD20EARS-00M',
    serial: '',
    removable: false,
    protocol: 'sata',
    group: '',
    device: '/dev/sdb'
  },
  {
    name: 'sdc',
    type: 'disk',
    fsType: '',
    mount: '',
    size:<PHONE_NUMBER>016,
    physical: 'HDD',
    uuid: '',
    label: '',
    model: 'WDC WD20EZRX-00D',
    serial: '',
    removable: false,
    protocol: 'sata',
    group: '',
    device: '/dev/sdc'
  },
  {
    name: 'loop0',
    type: 'loop',
    fsType: 'squashfs',
    mount: '/mnt/host/snap/core18/2721',
    size: 58314752,
    physical: '',
    uuid: '',
    label: '',
    model: '',
    serial: '',
    removable: false,
    protocol: '',
    group: ''
  },
  {
    name: 'loop1',
    type: 'loop',
    fsType: 'squashfs',
    mount: '/mnt/host/snap/core20/1822',
    size: 66355200,
    physical: '',
    uuid: '',
    label: '',
    model: '',
    serial: '',
    removable: false,
    protocol: '',
    group: ''
  },
  {
    name: 'loop2',
    type: 'loop',
    fsType: 'squashfs',
    mount: '/mnt/host/snap/core20/1852',
    size: 66392064,
    physical: '',
    uuid: '',
    label: '',
    model: '',
    serial: '',
    removable: false,
    protocol: '',
    group: ''
  },
  {
    name: 'loop3',
    type: 'loop',
    fsType: 'squashfs',
    mount: '/mnt/host/snap/lxd/24322',
    size: 117387264,
    physical: '',
    uuid: '',
    label: '',
    model: '',
    serial: '',
    removable: false,
    protocol: '',
    group: ''
  },
  {
    name: 'loop4',
    type: 'loop',
    fsType: 'squashfs',
    mount: '/mnt/host/snap/snapd/18357',
    size: 52260864,
    physical: '',
    uuid: '',
    label: '',
    model: '',
    serial: '',
    removable: false,
    protocol: '',
    group: ''
  },
  {
    name: 'loop5',
    type: 'loop',
    fsType: 'squashfs',
    mount: '/mnt/host/snap/snapd/18933',
    size: 55824384,
    physical: '',
    uuid: '',
    label: '',
    model: '',
    serial: '',
    removable: false,
    protocol: '',
    group: ''
  },
  {
    name: 'sda1',
    type: 'part',
    fsType: 'vfat',
    mount: '/mnt/host/boot/efi',
    size:<PHONE_NUMBER>,
    physical: '',
    uuid: 'DF3A-A56B',
    label: '',
    model: '',
    serial: '',
    removable: false,
    protocol: '',
    group: '',
    device: '/dev/sda'
  },
  {
    name: 'sda2',
    type: 'part',
    fsType: 'ext4',
    mount: '/mnt/host',
    size:<PHONE_NUMBER>48,
    physical: '',
    uuid: '06cee887-96fb-4bc2-89e3-a5d6eabb444b',
    label: '',
    model: '',
    serial: '',
    removable: false,
    protocol: '',
    group: '',
    device: '/dev/sda'
  },
  {
    name: 'sdb1',
    type: 'part',
    fsType: 'zfs_member',
    mount: '',
    size:<PHONE_NUMBER>720,
    physical: '',
    uuid: '10083813909764366566',
    label: 'pool_0',
    model: '',
    serial: '',
    removable: false,
    protocol: '',
    group: '',
    device: '/dev/sdb'
  },
  {
    name: 'sdb9',
    type: 'part',
    fsType: '',
    mount: '',
    size: 8388608,
    physical: '',
    uuid: '',
    label: '',
    model: '',
    serial: '',
    removable: false,
    protocol: '',
    group: '',
    device: '/dev/sdb'
  },
  {
    name: 'sdc1',
    type: 'part',
    fsType: 'zfs_member',
    mount: '',
    size:<PHONE_NUMBER>720,
    physical: '',
    uuid: '10083813909764366566',
    label: 'pool_0',
    model: '',
    serial: '',
    removable: false,
    protocol: '',
    group: '',
    device: '/dev/sdc'
  },
  {
    name: 'sdc9',
    type: 'part',
    fsType: '',
    mount: '',
    size: 8388608,
    physical: '',
    uuid: '',
    label: '',
    model: '',
    serial: '',
    removable: false,
    protocol: '',
    group: '',
    device: '/dev/sdc'
  }
]

Info output of dashdot cli
$ node dist/apps/cli/main.js info
INFO
=========
Yarn: 1.22.19
Node: v18.15.0
Dash: 5.1.3

Cwd: /app
Hash: 386191c474baca48d35d655bf77a8b4953e8645b
In Docker: true
In Podman: false
In Docker (env): true
Done in 0.28s.

What browsers are you seeing the problem on?
Chrome
Where is your instance running?
Linux Server
Additional context
The network and port part of my config may be confusing : this is because I am running dashdot with a reverse proxy container

Thanks for creating an issue. What RAID type is this running in?
Also, is there any way to get the current RAID type programmatically (through a CLI command) in ZFS?

What I am not understanding is - what is the real size of this pool? 1.8TB or 3.6TB? And what is the current usage you are seeing (in case it didn't really change since you opened this issue).
For the widget that would not be a problem, I already support RAID drives, I just need to know how I should parse what systeminformation is giving me.
So, what to do with those values:
const disks =  [
  {
    device: '/dev/sdb',
    type: 'HD',
    name: 'WDC WD20EARS-00M',
    vendor: 'Western Digital',
    size:<PHONE_NUMBER>016,
    bytesPerSector: null,
    totalCylinders: null,
    totalHeads: null,
    totalSectors: null,
    totalTracks: null,
    tracksPerCylinder: null,
    sectorsPerTrack: null,
    firmwareRevision: 'AB51',
    serialNum: '',
    interfaceType: 'SATA',
    smartStatus: 'unknown',
    temperature: null
  },
  {
    device: '/dev/sdc',
    type: 'HD',
    name: 'WDC WD20EZRX-00D',
    vendor: 'Western Digital',
    size:<PHONE_NUMBER>016,
    bytesPerSector: null,
    totalCylinders: null,
    totalHeads: null,
    totalSectors: null,
    totalTracks: null,
    tracksPerCylinder: null,
    sectorsPerTrack: null,
    firmwareRevision: '0A80',
    serialNum: '',
    interfaceType: 'SATA',
    smartStatus: 'unknown',
    temperature: null
  }
]
const sizes =  [
  {
    fs: 'pool_0',
    type: 'zfs',
    size:<PHONE_NUMBER>512,
    used: 131072,
    available:<PHONE_NUMBER>440,
    use: 0,
    mount: '/mnt/host/mnt/pool_0',
    rw: false
  },
  {
    fs: 'pool_0/media',
    type: 'zfs',
    size:<PHONE_NUMBER>336,
    used:<PHONE_NUMBER>96,
    available:<PHONE_NUMBER>440,
    use: 28.83,
    mount: '/mnt/host/mnt/pool_0/media',
    rw: false
  },
  {
    fs: 'pool_0/cloud',
    type: 'zfs',
    size:<PHONE_NUMBER>912,
    used:<PHONE_NUMBER>72,
    available:<PHONE_NUMBER>440,
    use: 10.05,
    mount: '/mnt/host/mnt/pool_0/cloud',
    rw: false
  }
]
const blocks =  [
  {
    name: 'sdb',
    type: 'disk',
    fsType: '',
    mount: '',
    size:<PHONE_NUMBER>016,
    physical: 'HDD',
    uuid: '',
    label: '',
    model: 'WDC WD20EARS-00M',
    serial: '',
    removable: false,
    protocol: 'sata',
    group: '',
    device: '/dev/sdb'
  },
  {
    name: 'sdc',
    type: 'disk',
    fsType: '',
    mount: '',
    size:<PHONE_NUMBER>016,
    physical: 'HDD',
    uuid: '',
    label: '',
    model: 'WDC WD20EZRX-00D',
    serial: '',
    removable: false,
    protocol: 'sata',
    group: '',
    device: '/dev/sdc'
  },
  {
    name: 'sdb1',
    type: 'part',
    fsType: 'zfs_member',
    mount: '',
    size:<PHONE_NUMBER>720,
    physical: '',
    uuid: '10083813909764366566',
    label: 'pool_0',
    model: '',
    serial: '',
    removable: false,
    protocol: '',
    group: '',
    device: '/dev/sdb'
  },
  {
    name: 'sdb9',
    type: 'part',
    fsType: '',
    mount: '',
    size: 8388608,
    physical: '',
    uuid: '',
    label: '',
    model: '',
    serial: '',
    removable: false,
    protocol: '',
    group: '',
    device: '/dev/sdb'
  },
  {
    name: 'sdc1',
    type: 'part',
    fsType: 'zfs_member',
    mount: '',
    size:<PHONE_NUMBER>720,
    physical: '',
    uuid: '10083813909764366566',
    label: 'pool_0',
    model: '',
    serial: '',
    removable: false,
    protocol: '',
    group: '',
    device: '/dev/sdc'
  },
  {
    name: 'sdc9',
    type: 'part',
    fsType: '',
    mount: '',
    size: 8388608,
    physical: '',
    uuid: '',
    label: '',
    model: '',
    serial: '',
    removable: false,
    protocol: '',
    group: '',
    device: '/dev/sdc'
  }
]

So I have digged around a bit, and apparently the way it works is the following:
ZFS can manage to put different file systems into a single pool, and to dynamically allocate and deallocate space between them. In my case, pool_0 contains two dataframes called cloud and media, who act like partitions, but for which the size is not fixed and can dynamically change.
There are basically 3 ways to check to storage state.
zpool list gives accurate info on the pool but nothing on the dataframes :
NAME     SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
pool_0  1.81T   726G  1.10T        -         -     0%    39%  1.00x    ONLINE  -

zfs list gives info on the pools and dataframes :
NAME           USED  AVAIL     REFER  MOUNTPOINT
pool_0         726G  1.05T       96K  /mnt/pool_0
pool_0/cloud   133G  1.05T      133G  /mnt/pool_0/cloud
pool_0/media   593G  1.05T      593G  /mnt/pool_0/media

We can see that both cloud and media have 1.05T available, but the total available space is 1.05T (so the real size of the pool is 1.8T and not 3.6)
Finally, I guess that systeminformation is based on the df command which is not very adapted to ZFS. Running it gives:
Filesystem     Type  1024-blocks      Used  Available Capacity Mounted on
pool_0         zfs<PHONE_NUMBER><PHONE_NUMBER>120       1% /mnt/pool_0
pool_0/media   zfs<PHONE_NUMBER> 622276352<PHONE_NUMBER>      36% /mnt/pool_0/media
pool_0/cloud   zfs<PHONE_NUMBER> 139146240<PHONE_NUMBER>      12% /mnt/pool_0/cloud

This output is wrong in every way possible.
As explained briefly here, df is unable to accurately read infos on zfs, and the reported usage can even change although the pool is inactive.
I think that the (current) output of systeminformation is not enough to accurately read the disk usage. I am not really sure if I should open an issue there, or if this is beyond the scope of the repo

I have created an issue on the systeminformation repo for this (https://github.com/sebhildebrandt/systeminformation/issues/802).
Please provide any requested information in case @sebhildebrandt needs it to develop this feature.
If this can be successfully implemented, I will start my implementation in Dashdot right after!

Also, can you please post the output of the command that gives the most information about the pool?
What I am after would be: disks in pool, size, usage, mounts.
 Is this on purpose, or by accident?
Also, could you maybe pipe it into a file, because I am not sure, but I think the separators are tabs, which would not be represented well in here.
zpool list -v -H -P > zpool_list_out.txt

And then upload that file to google drive, or our Discord server or anywhere where I can access it please.

Yes the separators do look like tabs and the first line indented separately. You can inspect the file here (Onedrive link)

Just as a note about Synology raid. Space used for the raid is all allocated to the first disk. All disks appear in Dash as size 0 except for the first disk, which shows only as the disk size (not the size of the raid). So in my case, my "space used" was actually more than the "total space available". This was ok for me though, I just hid all the disks except the first one and then used an override to set the correct total space.

As a ZFS user, I'm more interested in the state of my pools than the underlying drives other than if zpool status says something is wrong with a drive.

This is on my christmas list.


Update example to Obelisk v<IP_ADDRESS>

This gets the example running with Obelisk <IP_ADDRESS>
As it was, this would not build/run for me.

Thanks!


How to search files relative to the current buffer?

When using fzf, I have this line
nnoremap <leader>ff :Files %:p:h<CR>
and it lists all files in the current folder.
How can I achieve the same in vim-clap?
I tried
:Clap filer %:p:h<CR>
:Clap filer expand('%:p:h')<CR>
But manually typing directory name works as expected :Clap filer app
but it's not working. It opens search from the root project

filer also supports this. BTW the equivalent of Files is :Clap files not :Clap filer.

@liuchengxu How do I exactly use the new update?
:Clap filer %:p:h<CR>
:Clap filer expand('%:p:h')<CR>

Tried both but it's still not working. I've already updated my vim-clap

Well, works for me. Ensure you are using the latest commit https://github.com/liuchengxu/vim-clap/commit/b5b33d74bb6ae5568fd451476aed0a0ca0783e3e.

@liuchengxu Can you take a look at this asciinema I made? It works for the first time but it doesn't when tried for the second time.

This commit also adds the support for the other providers https://github.com/liuchengxu/vim-clap/commit/31d255eed79073d7409430b4456b40adc8179cd9.

@liuchengxu Just tested the latest commit but still not working as intended

Please file a new issue with reproduciable steps. With the video only, I have to spend more time to see what happens.


Testing Suite

Currently we don't have any automated testing to help protect against regressions. We should add tests.

@joshuapinter Are you still thinking about this? Would you be using Jest? I have no experience in testing, but I am interested in learning Jest and Enzyme. Maybe you can get something installed and configured the way you like and then I can help write tests?

Absolutely. I've been making improvements to Android recently, including SelectContact, which is really slick.
The more I think about testing, the more I think that Jest and Enzyme won't cut it. We use those in a project at work but mostly to render Components. In this library, we mostly need to test the access and response of the native Contacts on various device/OS combinations.
The only way I can think of to test this is through emulators, via Travis or CircleCI or something, where we can specify a testing matrix and confirm all supported OS and versions are passing the tests. Maybe we use Jest or something else for the actual tests but we'll need to have them run in an emulator to confirm they're working in the device environment. i.e. it can't just be standalone testing.
What do you think?

I think the two obvious ones are CircleCI and Travis CI.
Here is an example CircleCI project for React Native:
https://github.com/CircleCI-Public/circleci-demo-react-native
Might be a good starting place.
Plus, they offer free services for Open Source projects:

What if I am building open-source?
We offer a total of four free linux containers ($2400 annual value) for open-source projects. Simply keeping your project public will enable this for you!
We also offer the Seed plan free (at 1x concurrency) for OS X open source projects. Contact us at<EMAIL_ADDRESS>for access. If you are building a bigger open-source project and need more resources, let us know how we can help you!

Very interesting. I'd like to learn this stuff. It's a bit over me at the moment but I'm willing to follow along and help out here and there where I can. I agree that testing in a real/emulated device is superior as most of our problems have stemmed from weirdness on device.

Sounds good. Well, I'll get things setup and get the first couple tests done so that there is a pattern and let you finish it off from there.

Started playing around with this on the tests branch: https://github.com/joshuapinter/react-native-unified-contacts/tree/tests
Much more difficult than anticipated. Will require a whole weekend to get this working right so have to put this off until more time opens up.
It's important, though, so I'm keen to get it up and running eventually.

Awesome. Yep, there is only so much time in a week!


Events In PHP Script

Hello Guys,
I am in need of help in setting up the page with script in php and mysql in building the database. Read and re-read the small manual but found very confused and do not know what the structure of have to have for the events are recorded in the MySQL database and how the page code in PHP script. Can you send me or share with me the complete code ??? Thanks
This my code.

Twitter Bootstrap jQuery Calendar component

MEU CALENDARIO
";
echo "O QUE ISTO DA: ".$start."";
echo date("Y-m-d H:i:s");

$db    = new PDO('mysql:host=localhost;dbname=bootstrapcalendar;charset=utf8', 'root', 'XXXXXXX');
//$start = $_REQUEST['from'] / 1000;
//$end   = $_REQUEST['to'] / 1000;

$start = "1420890747263";
$end = "1420890747263";

$sql   = sprintf('SELECT * FROM events WHERE `datetime` BETWEEN %s and %s',
    $db->quote(date('Y-m-d', $start)), $db->quote(date('Y-m-d', $end)));

$out = array();
foreach($db->query($sql) as $row) {
    $out[] = array(
        'id' => $row->id,
        'title' => $row->name,
        'url' => Helper::url($row->id),
        'start' => strtotime($row->datetime) . '000',
        'end' => strtotime($row->datetime_end) .'000'
    );
}

echo json_encode(array('success' => 1, 'result' => $out));
exit;
?>

<!--<script type="text/javascript">

	var calendar = $('#calendar').calendar({
	events_source: [
    {
        "success": 0,
"error": "error message here"
    },
]});

</script>-->

MY DATABASE SQL
Table: events
Rows: datetime, id, name, datetime_end
Record: 2015-01-09 12:38:22, 1, User1, 2015-01-10 12:38:22
Thanks for help me

Twitter Bootstrap jQuery Calendar component

MEU CALENDARIO
";
echo "O QUE ISTO DA: ".$start."";
echo date("Y-m-d H:i:s");

$db    = new PDO('mysql:host=localhost;dbname=bootstrapcalendar;charset=utf8', 'root', 'XXXXXXX');
//$start = $_REQUEST['from'] / 1000;
//$end   = $_REQUEST['to'] / 1000;

$start = "1420890747263";
$end = "1420890747263";

$sql   = sprintf('SELECT * FROM events WHERE `datetime` BETWEEN %s and %s',
    $db->quote(date('Y-m-d', $start)), $db->quote(date('Y-m-d', $end)));

$out = array();
foreach($db->query($sql) as $row) {
    $out[] = array(
        'id' => $row->id,
        'title' => $row->name,
        'url' => Helper::url($row->id),
        'start' => strtotime($row->datetime) . '000',
        'end' => strtotime($row->datetime_end) .'000'
    );
}

echo json_encode(array('success' => 1, 'result' => $out));
exit;
?>

<!--<script type="text/javascript">

	var calendar = $('#calendar').calendar({
	events_source: [
    {
        "success": 0,
"error": "error message here"
    },
]});

</script>-->

Please, read this https://guides.github.com/features/mastering-markdown/, and you will know how to paste code here ;)
So, i edited your first comment for a better code display :-)
And now, for your code...
If you set an exit on php, that would.. well, exit, and stop executing. So, all the code (html, js, php... or whatever) that should be executed after exit, never happens.
The php code, should be appart, and thats the event source ;)

Not a calendar issue


Gardener 1.19

I think this is finally coming closer. @mwennrich Can you please update this PR to contain latest changes from master?

Superseded by https://github.com/metal-stack/gardener-extension-provider-metal/pull/203


Support for SET statements

#2724 mentioned that there's a need to support for arbitrary SET statements. Looking at the list here, many of those variables are pretty intrusive and likely to break Vitess. Just reserving a connection where you're allowed to set anything will not be sufficient.
So, I think we should instead support variables on a case-by-case basis. For some of them, special code may have to be written, like we had to do for autocommit.
Let's see if we can get the list built and addressed.

@sjmudd has a need for SET max_execution_time=120000. I told him this would probably be easy to add. Thoughts?
Additionally, some SET commands may be nice to just optionally no-op.

I guess coming from a setup where we knew we had a persistent connection then it makes sense to use some of these settings. That said by setting them one connection may behave differently to another and that would not be what's expected.  So sticky sessions would be quite nice to avoid this problem and possibly "reset state" when the app disconnects.
The other option might be (depending on the variables concerned) to accept the statement and ignore it, or give a warning. This allows for testing without breaking the app, but does not have the intended affect so might not be ideal but aprovides a stepping stone to play with vitess while the app is being adjusted.

But using something like the query hints in 5.7 seems better. https://dev.mysql.com/doc/refman/5.7/en/optimizer-hints.html#optimizer-hints-execution-time
as this works at the query level.  Potentially vitess could also understand the hints and even apply them even if the underlying mysql (mariadb ?) doesn't understand them. Maybe too much work though ?

There is functionality overlap between this variable and the vttablet query killer. The difference is that the query killer works for all queries including DMLs. So, this gives us a few options:

Re-interpret max_execution_time as applicable to all queries. If so, the implementation will just set the timeout on the query killer.
Perform more complex logic: Apply the timeout only if it's a SELECT query.
Keep the two functionalities orthogonal, and pass this timeout to the connection that would execute the statement, and reset it before returning to the pool.
Go the other way around: If it's a SELECT query, send the current timeout as max_execution_time to MySQL.

Option 1 would be the most elegant, but it's not 100% backward compatible.
2 & 3 differ from the fact that vttablet issues a KILL, which also kills the connection; It results in a reconnect later. But it's reliable and proven to work well in production.
To consider option 4, we have to see if max_execution_time is better than a KILL:

How reliable is it?
Does it kill the connection also?
What error do you get on killed queries?
Do all versions of mysql support it?

The initial use cases that this bug needed have been addressed. So, I'll close this. Going forward, we can create specific ones as needed.


Analog signal data not read from TDT SEV

Hi, all. I have never used neo directly before, so I apologize if this is just user error. But I have read through docs and code, and I think what I'm seeing is unexpected behavior.
I have a TDT tank containing a single TDT block using SEV (rather than TEV) format. I can successfully create a TdtIO object, read the .Tbk and .tsq files, which seem to reflect the right streams, with the right names, and the right numbers of channels. But it seems as if these header files are not being parsed properly, because the AnalogSignal objects that I get have incorrect sampling rates (1.0 Hz, vs the actual ~610Hz), incorrect durations (0.0s vs several hours), and no actual data (e.g. anasig.shape is (0, 4) for a 4 channel signal). I notice that the TdtIO._sigs_lengths dict is {0: {0: 0, 1: 0}}, which it seems should reflect the size of the data. But there doesn't at first glance appear to be anything wrong with the '.Tbk' and '.tsq' parsing, so perhaps the issue lies somewhere in between.
Here's a demonstration: read_tdt_using_neo.pdf. The data are ~800MB. Let me know if you'd like a copy in order to reproduce the issue.
Thanks,
Graham

Hhi Graham,
I did this reader very long time ago.
I don't remember any detail.
@JuliaSprenger : have refactor a bit recently. Maybe she will be able to help.

Hi @grahamfindlay, thanks for reporting this. My first suspicion would be that the .sev files are not detected by the io, as they are containing an additional _EEGf identifier in the filename, that is not present for the other files. Could you try renaming some of your sev files to just FCSV_EEG-220215-173221_Paul-220607-102425_Ch{1..4}.sev?

Thanks, both. 4 of the .sev files have _EEGf identifiers, and 4 have _EEGr identifiers. If I were to delete these identifiers from both sets of files, then the two sets would have identical filenames (i.e. the filenames are identical apart from these identifiers), so I can't do that. What I did instead, was:

Delete the _EEGf identifiers, and leave the _EEGr identifiers in place. This was unsuccessful (Sorry for the ugly pandoc/nbconvert formatting >.<)
attempt1.pdf

Delete the '_EEGf' identifiers, and delete the _EEGr files entirely. This was also unsuccessful.
attempt2.pdf

Perhaps it would be possible to use TDT's Python package to handle ingest of their data & headers/metadata, and massage this into the standardized neo format?

Update: Found another person with this more recent issue:


Update composer.json

~v2.0.0-rc1 is an illegal version constraint.

I had this issue today, too, but it seems to only affect older composer version. With the newest composer version, it is not an issue, but with the version from apt-get (Ubuntu Wily), it is.

Thank you very much @SvenRtbg, seems like this caused some confusion in the composer world. See: https://twitter.com/b_viguier/status/715108800290820096
Merged this commit with https://github.com/shopware/shopware/commit/10ef3c21c44df8f8ad44a1cc61f09d77765d7e26.


What is the variable name in the baselines folder?

Hi.
First of all, thank you for making the WeatherBench. This really helps my research having a benchmark for climate forecasting.
According to the data that you provided in https://dataserv.ub.tum.de/index.php/s/m1524895, I have a question that what is the variable name in the baselines folder. I tried using netCDF4 to read all .nc files, it turns out the parameter name is z (which is the geopotential). I think this geopotential might be 500 hPa geopotential (Z500)? I just want to make sure my understanding.
Best regards,
Ekasit

Hi, thanks for using the dataset. You can find a list of all the long and short names for the variables in the paper: http://arxiv.org/abs/2002.00469
So z is geopotential. On the download server there is a folder named geopotential_500 which contains only the Z500 field, and there is a geopotential folder which contains all vertical levels.
Hope this helps.


Exclude newly added failing validation runner tests

Closes #29972 ; #29973
Please add a meaningful description for your change here

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

R: @lostluck @kennknowles

validatesPortableRunnerBatch is still highly flaky, but
org.apache.beam.sdk.transforms.ReshuffleTest | PASSED | 7m 42.022s

org.apache.beam.sdk.transforms.ReshuffleTest
PASSED	7m 42.022s`


Use parent RamAccounting instance in ShardDMLExecutor

Summary of the changes / Why this improves CrateDB
The parent RamAccounting instance is either closed with a task or when
the final RowConsumer finishes.
Might fix flaky tests:
KillIntegrationTest
org.opentest4j.AssertionFailedError: [Query breaker not reset to 0 on node: node_s2]
  at __randomizedtesting.SeedInfo.seed([4C0486E0CAF108FE:C1CA1DD40B325C62]:0)
expected: 0L
  but was: 644245L

As seen in https://wacklig.pipifein.dev/github/crate/crate/5146a935-7e26-4ae6-9bc6-cfb21f003d07/01b148f2-8c1c-4051-a686-bbb92ebe2763
Checklist

[x] Added an entry in CHANGES.txt for user facing changes
[x] Updated documentation & sql_features table for user facing changes
[x] Touched code is covered by tests
[x] CLA is signed
[x] This does not contain breaking changes, or if it does:

It is released within a major release
It is recorded in CHANGES.txt
It was marked as deprecated in an earlier release if possible
You've thought about the consequences and other components are adapted
(E.g. AdminUI)

Do we want this in 5.1?

Do we want this in 5.1?

Maybe after we have verified that it fixes the flakyness


[work in progress] custom (composite) index

Description:
Custom index allows to combine attributes in index in custom order. [:a, :b, :c].
It allows queries by multiple attributes, such as: a or a&b, or a&b&c. This is a more performant way to query by multiple attributes than doing subqueries by a single attribute and then doing set intersection.
For the last attribute in a query, conditions can be in form of range, while preceding attributes can only be queried by value.
A model can have multiple custom indexes.
Current limitations:

types: Integer, Time, Nil
character limit of value length in decimal notation (currently 19)
exclusive ranges not supported

Example
class TestModel < T::Struct
  include Redcord::Base
  attribute :a, Integer, index: true
  attribute :b, T.nilable(Integer), index: true
  attribute :t, T.nilable(Time), index: true
  custom_index :first, [:a, :t]
  custom_index :second, [:a, :b, :t]
end

TestModel.where(a: val, t: interval).with_index(:first)
TestModel.where(a: val, b: val).with_index(:second)

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@169c881). Click here to learn what that means.
The diff coverage is 100.00%.

@@            Coverage Diff            @@
##             master      #72   +/-   ##
=========================================
  Coverage          ?   96.98%
=========================================
  Files             ?       31
  Lines             ?     1326
  Branches          ?        0
=========================================
  Hits              ?     1286
  Misses            ?       40
  Partials          ?        0

Impacted Files
Coverage Δ

lib/redcord/actions.rb
95.65% <100.00%> (ø)

lib/redcord/attribute.rb
97.05% <100.00%> (ø)

lib/redcord/redis.rb
100.00% <100.00%> (ø)

lib/redcord/relation.rb
98.55% <100.00%> (ø)

lib/redcord/serializer.rb
100.00% <100.00%> (ø)

spec/actions_spec.rb
95.45% <100.00%> (ø)

spec/migration/index_spec.rb
100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 169c881...66d5234. Read the comment docs.

class TestModel < T::Struct
  include Redcord::Base
  attribute :a, Integer, index: true
  attribute :b, T.nilable(Integer), index: true
  attribute :t, T.nilable(Time), index: true
  custom_index :first, [:a, :t]
  custom_index :second, [:a, :b, :t]
end

TestModel.where(a: val, t: interval, index: :first)
TestModel.where(a: val, b: val, index: :second)

I think we also want to clarify/verify: only the last attr in a custom index supports range queries
can we hash_func({a: int_val, b: float_val, c: string_val}) and format the hashed string? then we can simply the custom index into
<random_string>:formated_float/formated_int

reminder: we also need to update the vacuum logic to remove the entries from the custom_index sets

Have we considered a more implicit querying interface for custom indices, instead of having to specify with_index, or would this get too complicated? I think this would be more in line with ActiveRecord's interface

This would be ideal but I think we probably would want to build it in the next iteration. It is totally possible to determine which index to use: greedily pick the index that (validly) covers the most index attributes; then we can filter the rest in memory (in Ruby).


request for connection

I am interested in systems & rust, also an OL contributor. Would like to connect with you if you don't mind.

how do you like to connect?

how do you like to connect?

telegram if u prefer anonymity (https://t.me/samurai1024), or just email (yuanzhuoyang@gmail.com)?


Random "poisoned deferred unlock skipped"

I am randomly seeing the following message on multinode executions of HTR
[1 - 2000575df8b0]  542.495878 {4}{poison}: poisoned deferred unlock skipped - POSSIBLE HANG - lock=1f00010000000048

After I see this message the execution hangs.
How can I debug this issue?
@elliottslaughter Can you please add this issue to #1032 ?

Maybe some Realm people can comment. @eddy16112 @artempriakhin

Legion does do potentially poisoned reservation acquires/releases but they are always matched, meaning that if the acquire is poisoned then the release is poisoned too so neither of them should run. It shouldn't be possible to get an acquire that is not poisoned and release that is which could possible lead to a hang.

I should add though that this assumes that the poison propagates through a reservation acquire. If the poison doesn't propagate through a reservation acquire then the way Legion is doing things is not going to work.

Is there any update on this issue?

I added it to the Realm meeting agenda for next week.

I am going to take a look at it. I remember sending out a related patch to fix the following:

https://github.com/StanfordLegion/legion/issues/1610

Could be unrelated or could be the bug I introduced

Realm should't be acquiring the reservation when Reservation::acquire called with the poisoned precondition in the first place. Unless we are running into a race condition with the following:
      ...
      bool poisoned = false;
      if(wait_on.has_triggered_faultaware(poisoned)) {
        if(poisoned) {
          log_reservation.info()
              << "reservation:" << *this
              << " cannot be acquired due to poisoned precondition finish=" << wait_on;
          return wait_on;
        } else {
          Event e = get_runtime()->get_lock_impl(*this)->acquire(
              mode, exclusive, ReservationImpl::ACQUIRE_BLOCKING);
          log_reservation.info()
              << "reservation acquire: rsrv=" << *this << " finish=" << e;
          return e;
        }
        ...

@mariodirenzo Is that only with a multi-node setup? Would you be able to run whatever you are running with -level reservation=1?

I can't reproduce the issue on 1 node (at lease not deterministically as with 2 nodes).
The logs for a two node run are at rsv_0.log and rsv_1.log

@mariodirenzo Do you have logs on sapling2? Can you share the path perhaps instead of me downloading the log files.

Sure, /home/mariodr/public_html/rsv_*

@mariodirenzo Is this blocking you at the moment? We haven't made any progress since the last two weeks on this. Is that's an urgent matter we can priorize this on our side.
We may need to apply a custom patch with additional logging to root cause the problem. Do you have think we will able apply it and rerun?

The best would probably be for us to have a reproducer to run on sapling2

I'll try to reproduce the problem on sapling.
We can resort to applying the patch to the logging in case my attempts are not successful

I haven't been able to reproduce the issue on Sapling. I am unsure if this is system-related (though I can see it both on Lassen at LLNL and Leonardo at CINECA, which have different architectures) or simply because I am not running the calculation for long enough.
At this point, I would apply a custom patch to the setups on Lassen that reproduce the problem more regularly  and see if we can extract useful data out of the produced logs

@apryakhin What do we need to do for this? It's one of our last outstanding issues for the release.

@elliottslaughter Based on the data we have I can't confidently reason out where the problem is...at least in Realm. The presence of the "poisoned precondition" upon reservation release does not indicate the problem but merely states that the release of the reservation has been skipped (which must also mean that it wasn't acquired in the fist place).
There are two things we can do - we need either a reproducer on any system I can access in the form either realm, legion or htr test or a custom patch to collect more data for debugging.
I can prioritize working on this issue in next several days to make sure we have a root cause.

I don't know anything about the Leonardo system @mariodirenzo mentioned but I can tell you that getting access to LLNL systems will be slow (if it's possible at all). I believe @mariodirenzo was suggesting going the logging route.
Having said that, I wonder if it's worth just running the code longer on Sapling (possibly with some interfering processes) to see if we can make it happen there?

Thanks Elliot. I agree with your suggestion, but I believe @mariodirenzo has already attempted it without success.
I conducted further diagnostics on the logs and double-checked the data, leading to the following understanding:
We made approximately 6.7 million attempts to acquire roughly 16,000 different reservations over a certain period. As mentioned earlier, the "POSSIBLE HANG" message doesn't pinpoint the issue; it simply suggests that we skipped releasing the reservation, indicating a potential problem.
Here's an example of an invariant for a randomly selected reservation in the logs that remains consistent across all reservations
Number of reservation.acquire calls : 108883
Number of actual reservation acquires: 108612
Number of reservation.acquire dropped (poisoned) calls: 11
Number of reservation.release calls: 108894
Number of POSSIBLE HANGS..reservation.release dropped(deferred/poisoned) calls: 271
ASSERT(reservation.acquire(108883) + reservation.acquire_dropped(11) == reservation.release(108894))
ASSERT(reservation.acquire(108883) - reservation.actual_acquire(108612) == reservation.release_dropped(271))

On a per-reservation basis, we're encountering a certain number of skipped releases (indicated by the "POSSIBLE HANG" message), which indicates that we're deferring a release. This count amounts to 271, suggesting that we should also observe 271 fewer actual reservation acquisitions; otherwise, it could lead to deadlock. By subtracting the number of actual reservation acquisitions from the total attempts (108883 - 108612 = 271), we obtain the expected count of failed attempts. Notably, this count precisely matches the number of "POSSIBLE HANG" messages, which alerts us to the skipped reservation releases.
This holds true for each reservation in the logs provided here.
Unfortunately, this isn't a root cause and doesn't pinpoint the exact problem. However, it does somewhat increase confidence that we're not simply overlooking any unlock calls or mistakenly acquiring one reservation while performing a poisoned_release_skip on another.
I could be mistaken, so I'll need to discuss this with the team during today's meeting to ensure I haven't overlooked anything. My intuition suggests that we might be encountering an issue related to poisoned acquire/release semantics. I haven't delved into the exact event IDs used for deferred acquire/release, so there's a possibility something could be amiss there, although that would require a more detailed investigation.

Summarizing from our meeting today: we'll try to find someone with Lassen access to collaborate with @apryakhin so that we can reproduce this issue directly. @rohany volunteered to help out.

@muraj and @eddy16112 for visibility

@mariodirenzo - please provide instructions to reproduce this on lassen

I've already given the required files and instructions to @apryakhin via @rohany's account on Lassen

This issue has been open for a while now, and we need to reach some conclusion. We were able to set up a reproducer on Sapling (single node). The symptoms we observed are that the application does not make any progress. Specifically, it is expected to print the average value of the solution it is computing at every iteration. We did live debugging with GDB with @lightsighter on Sapling and can confirm that, whatever is going on, the runtime is doing some work. In other words, there is no hang.
The “POSSIBLE HANG” message reported at the beginning is unfortunately misleading, as it may report either the actual problem or false positives. In this case, it was the latter. I confirmed that by simply counting the number of lock/unlock operations on each reservation employed by Realm.
Now, two questions need to be answered. First, are the symptoms we see on Sapling the same as those that have been observed on Lassen (as initially reported)? In other words, there is no runtime hang while the application isn’t making any progress. Second, if that’s the case, why isn’t the application making progress? It seems that it has entered some form of loop, so we need to understand what it is doing.

@mariodirenzo To clarify a bit, it's very clear that Legion is actively running tasks for the application. Forward progress is being made. At no point does the application ever go into a hanging state. It's unclear to both me and Artem what is being done, but real application work is actively being accomplished. It may not be finishing for some reason at the application level, but it's not because it is hung. We need you to diagnose what is going on at the application level.


0.3 and beyond

Series of features I'm considering adding, not all may make themselves into versions
Profiler (Previously mentioned in #8 )
I've written a very basic profiler for CC. It would be great to expand this and integrate it with the debug wand.
Pocket computer notify (In #3)
As mentioned in this thread. Not sure how it would look but could be cool.
HTTP/Socket extensions
Mixture of OpenComputer's socket API and this topic. The issue with this is that scripts cannot rely on this existing so not sure it would get widespread use.
Turtle tool host (feature/tool-host Merged with #47)
Said this before, probably deleted the issue. Instead of the tool host providing an inventory slot though - it could just use the active slot of the turtle.
I'd probably have to add custom support for Bows and other tools that require right click.
This might be better implemented in a mod such as Peripherals++ instead.
Turtle wand.
I've always liked the idea of using Turtle labels to convey information to the player. In CC 1.6.0, turtle names were displayed even when not hovering over. The turtle wand could implement similar rendering functionality to Ender IO's 'Staff of Travelling' - drawing turtles larger than they are, through walls, with their label shown. Probably at the same time as I do #9.

I got something working:

I would love to see the Pocket computer notify as well as a socket Api which resembles OCs Api (especially the TCP socket ;) )
I think the 'issue' that scripts can not rely on this existing isn't that bad, since CC has the http api and the pastebin program which noone can rely on either, since they can be disabled in the configs.
I just wanted to give my two cents to this update, which I will look forward to

Implemented socket/TCP API in 056d97c0fae98194807a726eed2924fc752ba9df. I'd love to implement a pocket computer notification system, but I'm not sure of the best look for it. Thoughts/feedback welcome.


Feasibility of Using Library With Just Unique Usernames

Hello,
I recently stumbled upon this project and it looks fantastic, and I would really like to use this for a project I am working on. However, for usernames instead of the username being email-based, it's just based upon a unique username in my project. Hypothetically speaking, if I wanted to fork and tweak the library under the hood. How difficult would it be to have it so I have the option of having usernames upon email or unique usernames?

Hi there,
This is not something that's supported, and I don't wish to support it for now.
However, it should indeed be quite straightforward to do it in a fork. Here is the plan:

Add username property to the User models.
Tweak create_user function to check username unicity.
Add a get_by_username method in DB classes.
Tweak authenticate method in BaseUserDatabase to login with a username.

I think you should have most of it working doing this.
Cheers!


Add trajectory selection

Didn't @royayal1 and Adi decided no trajectory is needed?

Didn't @royayal1 and Adi decided no trajectory is needed?

They said there is no need in complex trajectory(path weaver/planner), but there's still would be at least 2.

Didn't @royayal1 and Adi decided no trajectory is needed?

They said there is no need in complex trajectory(path weaver/planner), but there's still would be at least 2.

Actually it's exactly two, @guy-av   They're what i sent on the whatsapp gc


Atom is now missing a few heavy plug-ins

atom is now missing some of the heavyweight plug-ins, like the excellent IDE plug-ins.
Officials can organize a group of active users in the open source community to solve this problem

Thanks for reaching out!
We have passed along the information to the maintainers team. Because we treat our issues list as the Atom team's backlog, we close feedback issues after passing along the information to the maintainers to keep our backlog clean and focused. In the future, if you want to send feedback to the maintainers you can do so more directly by sending email to<EMAIL_ADDRESS>


Feature/generic data upload worker

Depends on https://github.com/ClimateWatch-Vizzuality/climate-watch-gems/pull/8 to simplify code (by removal of repetitive code)
Update:

Updated Gemfile with new version of cw data uploader gem
Change deployment scripts with referral to new S3_BUCKET_NAME and new secret vars (the same ones as in South Africa platform)
Add new CW_FILES_PREFIX env variable (needs to be set both in production and staging as they will be different, they represent the path to folders in s3 bucket), i.e. folder_in_bucket/staging/climate-watch/ -> always with a slash in the end ❗
Add missing socioeconomics page to admin panel

thanks! updated the gem

Hi guys, I see a conflict here. Can someone have a look, please? Thanks!

I resolved the conflict and added CW_FILES_PREFIX to deployment scripts like in South Africa

I will copy the files into the new bucket now


Fix couchdb backup endpoint; Set ENABLE_ZABBIX to FALSE by default

The couchdb backup endpoint was wrong (missing underscores in URL)
The ENABLE_ZABBIX was set to TRUE from the base image, which will cause some errors get logged (not a big deal)

Thanks!


SIGSEGV on connect

Here is a minimized testcase that causes uring to segfault:
let rec consume uring =
  match Uring.wait ~timeout:1.0 uring with
  | None ->
      ()
  | Some _ ->
      consume uring

let sockets =
  Array.init 500 (fun _ ->
      let s = Unix.socket Unix.PF_INET Unix.SOCK_STREAM 0 in
      (s, Unix.ADDR_INET (Unix.inet_addr_of_string "<IP_ADDRESS>", 631)) )

let uring = Uring.create ~polling_timeout:1000 ~queue_depth:128 ()

let () =
  let () =
    sockets
    |> Array.iter
       @@ fun (sock, dest) ->
       let rec loop () =
         match Uring.connect uring sock dest 5 with
         | Some _ ->
             ()
         | None ->
             let (_ : 'a Uring.completion_option) =
               Uring.wait ~timeout:1.0 uring
             in
             loop ()
       in
       loop ()
  in
  while Uring.active_ops uring > 0 do
    consume uring
  done

Compile and run like this (assuming you have something listening on port 631 localhost, if not pick another port...):
$ ocamlfind ocamlopt -package unix,uring conn.ml -linkpkg -o conn
$ ./conn
[1]    112639 segmentation fault (core dumped)  ./conn

Software and hardware environment:
$ ocamlopt --version
5.0.0
$ uname -a
Linux fedora38.fedora 6.2.14-300.fc38.x86_64 #1 SMP PREEMPT_DYNAMIC Mon May  1 00:55:28 UTC 2023 x86_64 GNU/Linux
$ inxi
CPU: 16-core AMD Ryzen 9 7950X (-MT MCP-) speed/min/max: 3131/3000/5880 MHz
Kernel: 6.2.14-300.fc38.x86_64 x86_64 Up: 10h 4m Mem: 5939.9/63423.8 MiB
(9.4%) Storage: 24.99 TiB (4.0% used) Procs: 536 Shell: Zsh inxi: 3.3.26

What I tried:

changing the number of sockets from 500 to 50: no crash
removing the '~timeout' flag from Uring.wait: no crash, but stuck
removing the ~polling_timeout flag: no crash, but stuck
using the debug runtime: didn't provide more info about the crash

coredumpctl gdb:
Core was generated by `_build/default/loadtest_uring.exe http://localhost:631'.
Program terminated with signal SIGSEGV, Segmentation fault.

warning: Section `.reg-xstate/113639' in core file too small.
#0  0x00000000004c7872 in ocaml_uring_wait_cqe_timeout (v_timeout=<optimized out>, v_uring=<optimized out>) at uring_stubs.c:688
688         id = (long)io_uring_cqe_get_data(cqe);
[Current thread is 1 (Thread 0x7f17e38da780 (LWP 113639))]
(gdb) bt
#0  0x00000000004c7872 in ocaml_uring_wait_cqe_timeout (v_timeout=<optimized out>, v_uring=<optimized out>) at uring_stubs.c:688
#1  <signal handler called>
#2  0x0000000000453007 in camlUring__fun_3000 () at uring.0.5/lib/uring/uring.ml:502
#3  0x0000000000452e15 in camlUring__fn_on_ring_2135 () at uring.0.5/lib/uring/uring.ml:484
#4  0x0000000000452f81 in camlUring__wait_2146 () at uring.0.5/lib/uring/uring.ml:502
#5  0x000000000044fc38 in camlDune__exe__Loadtest_uring__loop_589 ()
#6  0x0000000000487865 in camlStdlib__Array__iter_329 () at array.ml:92
#7  0x000000000044fd1a in camlDune__exe__Loadtest_uring__entry ()
#8  0x000000000044c940 in caml_program ()
#9  <signal handler called>
#10 0x00000000004f86cb in caml_startup_common (pooling=<optimized out>, argv=0x7ffe5f990e88) at runtime/startup_nat.c:129
#11 caml_startup_common (argv=0x7ffe5f990e88, pooling=<optimized out>) at runtime/startup_nat.c:85
#12 0x00000000004f875b in caml_startup_exn (argv=<optimized out>) at runtime/startup_nat.c:136
#13 caml_startup (argv=<optimized out>) at runtime/startup_nat.c:141
#14 caml_main (argv=<optimized out>) at runtime/startup_nat.c:148
#15 0x000000000044c52c in main (argc=<optimized out>, argv=<optimized out>) at runtime/main.c:37

(gdb) print cqe
$1 = (struct io_uring_cqe *) 0x0

The following change avoids the crash, but I don't know if it is correct (program still stuck, so there are likely still bugs in the way I've written it):
git diff
diff --git a/lib/uring/uring_stubs.c b/lib/uring/uring_stubs.c
index 8e050dc07e..ef02b49008 100644
--- a/lib/uring/uring_stubs.c
+++ b/lib/uring/uring_stubs.c
@@ -687,6 +687,8 @@ value ocaml_uring_wait_cqe_timeout(value v_timeout, value v_uring)
       unix_error(-res, "io_uring_wait_cqe_timeout", Nothing);
     }
   } else {
+    if (!cqe)
+      CAMLreturn(Val_cqe_none);
     id = (long)io_uring_cqe_get_data(cqe);
     io_uring_cqe_seen(ring, cqe);
     CAMLreturn(Val_cqe_some(Val_int(id), Val_int(cqe->res)));

(if I change number of sockets from 500 to 50 it doesn't stuck, doesn't crash and finishes).
In general the C stub appears to be very unsafe, and I think it'll take a lot more than the above change to make it safe even on a single threaded, one domain application (e.g. passing a Sockaddr_val pointer to 'io_uring_prep_connect' which just puts the pointer into the ring for the kernel to read it later. But the OCaml GC may move that value from the minor heap to the major heap, causing the pointer to point to stale data, etc.)

In general the C stub appears to be very unsafe, and I think it'll take a lot more than the above change to make it safe even on a single threaded, one domain application (e.g. passing a Sockaddr_val pointer to 'io_uring_prep_connect' which just puts the pointer into the ring for the kernel to read it later. But the OCaml GC may move that value from the minor heap to the major heap, causing the pointer to point to stale data, etc.)

Looking at 'uring.ml' it does seem to protect against this with 'Heap.alloc/Heap.free' and putting the sockaddr into 'extra_data', so the GC doesn't free the sockaddr while the entry is in use.

With the following change to my testcode (and the above uring bugfix) the code no longer crashes or hangs,
basically it needed a connect timeout, because the server on port 631 stopped accepting connections after a while:
let rec consume f uring =
  if Uring.active_ops uring > 0 then
  match Uring.wait ~timeout:5.0 uring with
  | None ->
      ()
  | Some {result;data} ->
      f result data;
      consume f uring

let sockets =
  Array.init 500 (fun _ ->
      let s = Unix.socket Unix.PF_INET Unix.SOCK_STREAM 0 in
      (s, Unix.ADDR_INET (Unix.inet_addr_of_string "<IP_ADDRESS>", 631)) )

let uring = Uring.create ~polling_timeout:1000 ~queue_depth:1000 ()

let () =
  let jobs = Array.make (Array.length sockets) None in
  let () =
    sockets
    |> Array.iteri
       @@ fun i (sock, dest) ->
       let rec loop () =
         match Uring.connect uring sock dest i with
         | Some _ as job ->
            jobs.(i) <- job
         | None ->
             match Uring.wait ~timeout:1.0 uring with
             | None -> loop ()
             | Some { result; data } ->
                jobs.(data) <- None;
                loop ()
       in
       loop ()
    in
    let () = uring |> consume @@ fun result data ->
        jobs.(data) <- None
    in
    if Uring.active_ops uring > 0 then
        let () = jobs |> Array.iteri @@ fun i job ->
        match job with
        | None -> ()
        | Some job ->
            jobs.(i) <- Uring.cancel uring job i
        in
        uring |> consume @@ fun result data ->
            jobs.(data) <- None

I opened a PR with the fix (it'd be great if the above testcase could be converted to a stress test, perhaps with a listening socket spawned as part of the test using 'socketpair', but I ran out of time to do that right now.)


Remove Babel hard dependency in setup.py

Babel is only needed as a tool to build the distribution. That can be done in a separate packaging step.
Fixes #46

Current coverage is 99.66%

Merging #47 into master will not affect coverage as of c45ff0c

@@            master     #47   diff @@
======================================
  Files           14      14
  Stmts          598     598
  Branches         0       0
  Methods
======================================
  Hit            596     596
  Partial          0       0
  Missed           2       2

Review entire Coverage Diff as of c45ff0c

Powered by Codecov. Updated on successful CI builds.


Unable to generate all resources (for methods with brackets)

Unable to generate all resources (for methods with brackets) for example https://github.com/mulesoft/raml-for-jax-rs/blob/master/raml-to-jaxrs/examples/jersey-example/src/main/resources/raml/sales-enablement-api.raml
Output:
Generating resources
Failed to generate sub method: /presentations/{presentationId}
because the sub method name could not be resolved
Failed to generate resources for uri: /presentations/{presentationId}
because this route doesn't have any methods or sub resources
✔ PresentationApi
Failed to generate sub method: /products/{productId}
because the sub method name could not be resolved
✔ ProductApi

+1

Any update on this?

+1


Units of laser_label

Is the positions of box in the dataset (laser_labels - center_x,y,z, and width, length, height) written in meters? If so, are speed_x, y and accel_x,y  values in m/s in global coordinate system? I couldn't find their units.

Yes that's correct, everything should be in SI units.

Thank you for answering! I'm not sure if you might be able to answer this question, but when I extract point cloud from range image, like
'points_xyz': array([[-46.842773  ,  10.449219  ,   4.1116333 ],
[-67.19141   ,  27.495117  ,   5.1240845 ],
are they xyz values in lidar sensor coordinate?

Hi! If you're following our provided code to convert the range image into a point cloud, these coordinates will be in vehicle coordinate system. In particular, please refer to https://github.com/waymo-research/waymo-open-dataset/blob/bae19fa0a36664da18b691349955b95b29402713/waymo_open_dataset/utils/range_image_utils.py#L610 for details.


April 2022 Release Endgame

April 2022 Release Endgame
Branch: https://github.com/microsoft/azuredatastudio/tree/release/1.36
Testing Links:

Windows User EXE
Windows System EXE
Windows ZIP
MacOS Zip
Linux tar.gz
Linux rpm
Linux deb

Install the latest build of ADS from the link above.
Scenario Sign Off
Scenario Sign offs have been moved to this page
Partner Sign-offs

[ ] PGSQL @QingqingYuan2
[ ] SQL On Demand @BranislavGrbicMDCS
[ ] SQL Assessment @alexprotsenko

Release completed (and Hotfixed)


Show sender name in Bubble

How can i customize the message bubble in order to show the sender name?

Yes, I need this too...

If you want to put the name above the message you can use a decorator view. Otherwise you need to create your own bubbles. We don't aim to have super customizable ChattoAdditions framework with all the UI requirements that you request - that just doesn't scale and won't satisfy everyone anyways.
So my advice is to create your own ChattoAdditions framework with your own bubbles UI. If then you want to share your Additions framework with the rest of the community then that would be very nice :)

Diego, could you describe decorator view idea in several words, please?

Does anybody solved that problem?


added --ignore-installed flag to pip install

fixes #10
added --ignore-installed flag to pip install

--ignore-installed will just overwrite the existing with the current version, but will not remove files that were deleted in the update.

thank you @twinstar26


Update GitHub Actions Workflow for Compatibility and Dependency Fixes

Type of change
Pick at least one.

[X] Bug fix (non-breaking change that fixes something)
[X] Code quality improvement (refactor, performance improvements)

Breaking change
Does this Pull Request introduce a breaking change?
No
Proposed change
This PR addresses the following:

Updates actions to v4 as v3 actions will be deprecated soon: https://github.blog/changelog/2024-04-16-deprecation-notice-v3-of-the-artifact-actions/
Installs wireless-regdb_2024.10.07-1_all because wireless-regdb_2024.07.04-1_all is no longer available and 404s breaking the build everytime
Updates the method to get output from docker as set-output is deprecated: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
Replaces the dependency on archived marvinpinto/action-automatic-releases@latest with a fork joshschmelzle/action-automatic-releases@1c60c2dcfeb5382b81099310bdb4a376dc7753a0 which addresses set-output deprecation

Testing
Please explain how you tested this change manually, and if applicable, what new tests you added.
Tested in fork to confirm images were built and uploaded. Please squash merge this PR.
Checklist

[X] I have read the contribution guidelines and policies
[X] I have targeted this PR against the correct git branch
[ ] I ran the test suite and verified it succeeded
[X] I linked an approved GitHub issue to this PR (in the next section). No PR without a related GH issue. Conversations about your PR efforts in other channels such as electronic mail, social media, morse code, and WLAN Pi Slack do not count.
[ ] I have updated the changelog (if applicable)
[ ] I have added or updated the documentation (if applicable)

Related Issues/PRs
Pick at least one.
This PR fixes #63 and #40


Update readme to check that it is working

currently it is mentioning to test on a file without tname. An adoc file must be used (or the asciidoc text mode but too much complicated). i think the readme should be updated to mention the adoc file creation

done with https://github.com/abhatt-rh/asciidoc_vscode_snippets/pull/10


[FR] - Add a Cardano-CLI function to expose the calcMinimumDeposit calculation for NativeAssets

External
Area
Native Tokens    Related to Native Tokens (Mary).
Describe the feature you'd like
We need a function to calculate the right minDeposit (minAda) for a --tx-out transaction build-raw with cardano-cli that includes NativeAssets. In a perfect world we would provide the cardano-cli the same --tx-out string that we will use in the build-raw command and get back the minDeposit (minAda) amount for that --tx-out.
Like:
cardano-cli transaction calculate-min-deposit --tx-out TX-OUT --protocol-params-file FILE [--out-file FILE]
Currently its just guessing or in the solution for my StakePoolOperator-Scripts, Jormanager and CNTools its just minUTXOvalue (1 ADA) + 1 ADA per asset in the output. That works, but is way to much. So we need a workable method to get the right value to send out NativeAssets with the minimum needed Deposit.
Related:
https://github.com/input-output-hk/cardano-node/pull/2325
https://github.com/input-output-hk/cardano-node/issues/2298
@Jimbo4350

I have now written a working bash-function for that. It takes the JSON of the protocol-parameters and also the tx-out string you wanna use. It returns the minAda needed to do that. Ok, so currently no rush on that cardano-cli function i guess, but if you find the time to integrate it, would be nice.

Maybe something like this?
$ CARDANO_NODE_SOCKET_PATH=mainnet.socket cardano-cli transaction calculate-min-value --mainnet --multi-asset "5<PHONE_NUMBER>0000000000000000000000000000000000000000000000.couttscoin + 6<PHONE_NUMBER>0000000000000000000000000000000000000000000001.jimbocoin"
Result Lovelace 1703702

Done


The Bert model

May I ask if you used the training model of 'DNABert' directly or fine-tuned it?
I see  that you are using the model directly in the code, but in the article you say fine-tuning it.

We open source the fine-tuned bert model, since it is fine-tuned based on the DNABERT model, so the number of parameters is consistent.

OK，thank you for your reply！


Orcid login

[x] Configure ORCID login properly
[x] Generate jwt tokens to user logged in through ORCID

Fixes #509

Downgrading all-auth at most up to date version is causing issues
https://stackoverflow.com/questions/78477908/dj-rest-auth-with-google-login-typeerror-oauth2provider-get-scope-takes-1-po

Changing the callback to sample one that does not do anything, makes it so that the error "Failed to exchange code for access token" does not occur anymore whe we call /dj-rest-auth/orcid/ with the retrieved code the first time - it gives us the access token and all other info - including if the user has an email associated to him or not.

Sending the code a second time give us the error "Failed to exchange code for access token" likely as the code as already been "spent"

Currently trying an implementation with  only all-auth
Managed to find the path to authenticate - https://docs.allauth.org/en/latest/headless/openapi-specification/#tag/Authentication:-Providers/paths/~1_allauth~1browser~1v1~1auth~1provider~1redirect/post - it returns a code - althought it seems its not fully doing his job since when I try to get the user via https://docs.allauth.org/en/latest/headless/openapi-specification/#tag/Authentication:-Current-Session I get 401 meaning authentication has not been completely done.
Managed to find the path to send the email - https://docs.allauth.org/en/latest/headless/openapi-specification/#tag/Authentication:-Providers/paths/~1_allauth~1{client}~1v1~1auth~1provider~1signup/post
Next Actions:
Understand whats going on with the callbacks as it seems that they are the sources of the issue

Facts:

dj-rest-auth seems to be skipping the verify email is there step present in all-auth.
It seems that function process_signup is not being entered (link) to be understood why

Next Steps:

Debup step by step whats happening during the /dj-rest-auth/orcid to confirm it is indeed whats happening.
Some Possible Implementations:

Implement our own callback view tha does the email check for us
Implement it using only all-auth headless - (will require meddling w/ JWT Auth as its not implemented by default  )

Using simply dj-rest-auth out of the box is not recommended to orcid as the Callback is not implemented

Possible Implementations:
a) Implement our own custom callback view - Based it on OAuth2CallbackView
b) Implement it using only all-auth headless - (will require meddling w/ JWT Auth as its not implemented by default  )
Trying option b)

@DonHaul there is a conflict here with poetry


Adding basic rules for AsyncAPI

Reason/Context
This ruleset should support both OpenAPI and AsyncAPI linting for ease of use
Description
Basic rules should be included for AsyncAPI such as:

Checking examples are provided,
Checking examples are valid against the schema,
Checking parameters/messages example names are matching to get complete mocks.

Implementation ideas
No response

This can only work using Spectral. Vacuum does not support AsyncAPI parsing at the moment.


Thread safe variables/event properties

NLog version: 4.7
Platform: .Net 4.5
Like many people on the net I am using LogManager.Configuration.Variables to log some common information in each line of a webservice log.
However, I have noticed that it is not thread safe, and it is possible for a thread to overwrite a previous threads variables before the previous thread is finished logging.
I have looked into EventProperties, but there is a warning that SetProperty is similarly not thread safe. It would be good to have the ability to easily set some variables/properties at the start of the request such that those variables can be accessed by all loggers in that thread for the rest of that request's lifetime.

Actually, looks like https://github.com/NLog/NLog/wiki/MDLC-Layout-Renderer
Is exactly what I am after, this definitely needs to be publicized better - I would argue that all LogManager.Configuration.Variables should operate like MDLC

Please note that logger.WithProperty is also thread safe. I would recommend to use that over the mdlc, if suitable/possible


Transferring progress bar sometimes only shows in the app's icon in the taskbar, and not in the app itself where it also should.

Have you noticed when does it work and when it doesn't?
Are you talking about the total progress bar above the address / navigation box?
What about progress in the file operations view? Does it work and does it display more information apart from percentage?

-No
-Yes
-It does, the progress bar in that works, but the colour of the filled part and the not filled part of the progress bar there is the same and you can only see the progress when you hover over the bar and the not filled part gets lighter in colour.

https://github.com/user-attachments/assets/fc5c12fc-9471-48eb-ab20-c580e20a228a

Ok, I see the problem - your system accent color is gray.
I expected people to use different accent colors, but I didn't think about gray 😅.
Sadly there's a lot of dependency on the system accent color.
I might add an option to use a color different than the system one, but not in the upcoming version.

oh got you


Use jobset-system instead of kind-system for jobset

We are not finding the correct namespace so we have no logs.
Fixes #356

Thanks for looking into this
/lgtm
/approve

/hold
Want to confirm that it works before merge.

Can confirm that this works: https://gcsweb.k8s.io/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_jobset/358/pull-jobset-test-e2e-main-1-28/1737556386608320512/artifacts/
You can browse there and see the logs.
/hold cancel

/retest

/retest


New CosmPy release

Make a new CosmPy release. Also test the automated release process as part of it.

0.6.1 is out


auto sync master and master-v2

Probably reuse the GH action setup in plasma-contracts and plasma-contract-private

What should be the ideal way to merge into master-v2?
If we squash-and-merge the commits apparently add up every-time (since there is no reference to this older commits on master-v2)
For instance, the successive pull requests will look like this-
1st PR

2nd PR

3rd PR


feat: add function to run spanner emulator with returned information

Still relevant?

@sfinnman :point_up:


Got code 88, Rate limit exceeded just by twitter = TweeterPy()

hey pal, it's me again.
I've encountered this since last tuesday, when I try to initialize the crawler script which runs perfect for like whole month.
from tweeterpy import TweeterPy
twitter = TweeterPy()

it says
2024-03-19 02:41:31,223 [ERROR] :: Couldn't generate a new session.
'guest_token'
Traceback (most recent call last):
  File "/home/yitinglin/metacrm-twitter-crawler/tweeterpy/tweeterpy.py", line 159, in generate_session
    guest_token = make_request(
KeyError: 'guest_token'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/yitinglin/metacrm-twitter-crawler/tweeterpy/tweeterpy.py", line 29, in __init__
    self.generate_session()
  File "/home/yitinglin/metacrm-twitter-crawler/tweeterpy/tweeterpy.py", line 159, in generate_session
    guest_token = make_request(
KeyError: 'guest_token'

when I dig into the source code and print out the result of make_request to the guest token, I found that I got
{'code': 88, 'message': 'Rate limit exceeded.'}

I'm wondering if you have encountered this issue before, on the guest token request.

Hmm that's strange.
Have you tried doing it from some other machine and with a new IP address. I think the limit is related to the IP address (If you are scraping continuously). Try connecting to some other network or maybe something like a VPN/Proxy.
Let me know how it goes.

is it possible to extract all the followers of a page with (5M followers)? i tried multiple ways. pagination=True, and pagination = False. it doesn't seem to work. tried changing auth_token and proxies after failure. still facing too many requests

Hello, i am facing a problem with your code when using threads. i tried multiple thread types. tried to put each instance of your Twitter instance in a separated instance but it's not working for me. whith 1 thread it works completely fine but when i am using multi threads even if each instance has it's own proxy rotating every request.  it's still throwing 403 forbidden when calling your login function. is there a solution ? here's my code
list_tokens = open('twitter_tokens.txt', 'r', encoding='utf-8').read().splitlines()
profiles_to_scan = open('profiles_to_scan.txt', 'r', encoding='utf-8').read().splitlines()
all_proxies = open('proxies_crack.txt', 'r', encoding='utf-8').read().splitlines()
def getProxy():
return next(proxy_pool)
class Proxy :
def init(self,proxy):
self.user = proxy.split(':')[2]
self.password = proxy.split(':')[3]
self.ip = proxy.split(':')[0]
self.port = proxy.split(':')[1]
proxy_pool = itertools.cycle(all_proxies)  # Create a cycle iterator from the proxy list
accounts_pool = itertools.cycle(list_tokens)
def get_account():
return next(accounts_pool).strip()
class Profile():
 def __init__(self,profile):
    self.profile = profile

 def get_followers(self):
    proxy_object = Proxy(getProxy())
    proxy_url = "http://" + proxy_object.user + ":" +proxy_object.password + "@" + proxy_object.ip + ":" + proxy_object.port + "/"

    proxies_formatted = {
                "http":  proxy_url,
                "https":  proxy_url,
            }

    config.PROXY = proxies_formatted

    twitter = TweeterPy()
    twitter.generate_session(auth_token=get_account())
    self.profile = self.profile.strip()
    has_more = True
    cursor = None
    while has_more:
        try:
            response = None
            response = twitter.get_friends(self.profile,follower=True, end_cursor=cursor,pagination=False)
            with open(self.profile+'.txt', 'a',encoding='utf-8') as save_followers:
                    for follower in response['data']:
                        screen_name = follower['content']['itemContent']['user_results']['result']['legacy']['screen_name']
                        save_followers.write(screen_name+'\n')

            has_more = response.get('has_next_page')
            api_rate_limits = response.get('api_rate_limit')
            limit_exhausted = api_rate_limits.get('rate_limit_exhausted')
            if has_more:
                cursor = response.get('cursor_endpoint')
            ## YOUR CUSTOM CODE HERE (DATA HANDLING, REQUEST DELAYS, SESSION SHUFFLING ETC.)
            ## time.sleep(random.uniform(7,10))
            if limit_exhausted:
                raise RateLimitError
        except Exception as error:
            print(error)
            twitter.generate_session(auth_token=get_account())
            config.UPDATE_API

def create_and_launch_threads(profile):
profile_client = Profile(profile)
profile_client.get_followers()
return
with concurrent.futures.ThreadPoolExecutor(max_workers=2) as executor:
# Submit the function to the executor for each number
futures = [executor.submit(create_and_launch_threads, profile) for profile in profiles_to_scan]
# Wait for all futures to complete and get their results
results = [future.result() for future in concurrent.futures.as_completed(futures)]

Hey @ilonabehn2
Could u please attach the logs? Make sure to strip off any sensitive data.
I will take a look at the code in the meantime.
Thanks

now it's a different error. it runs properly and after a while it start throwing this.

right now i am using a single thread in multiple cmd's. but still facing errors even without proxies.  it got around 50k followers and then breaks out of the loop see:

i am using 1000 auth_tokens. the data is stripped for sure. and here's my code.:
from tweeterpy import TweeterPy
import sys
from tweeterpy.util import RateLimitError
import itertools
import urllib3
#from time import sleep
urllib3.disable_warnings()
#import concurrent.futures
list_tokens = open('twitter_tokens.txt', 'r', encoding='utf-8').read().splitlines()
accounts_pool = itertools.cycle(list_tokens)
def get_account():
return next(accounts_pool).strip()
class Profile():
 def __init__(self,profile):
    self.profile = profile

 def get_followers(self,cursor=None):

    twitter = TweeterPy()
    twitter.generate_session(auth_token=get_account())
    self.profile = self.profile.strip()
    has_more = True
    #cursor = None
    while has_more:
        try:
            response = None
            response = twitter.get_friends(self.profile,follower=True, end_cursor=cursor,pagination=False)
            with open(self.profile+'.txt', 'a',encoding='utf-8') as save_followers:
                    for follower in response['data']:
                        screen_name = follower['content']['itemContent']['user_results']['result']['legacy']['screen_name']
                        save_followers.write(screen_name+'\n')

            has_more = response.get('has_next_page')
            api_rate_limits = response.get('api_rate_limit')
            limit_exhausted = api_rate_limits.get('rate_limit_exhausted')
            if has_more:
                cursor = response.get('cursor_endpoint')
            ## YOUR CUSTOM CODE HERE (DATA HANDLING, REQUEST DELAYS, SESSION SHUFFLING ETC.)
            ## time.sleep(random.uniform(7,10))
            if limit_exhausted:
                raise RateLimitError
        except Exception as error:
            print(error)
            twitter.generate_session(auth_token=get_account())
            return self.get_followers(cursor=cursor)

def create_and_launch_threads(profile):
profile_client = Profile(profile)
profile_client.get_followers()
return
create_and_launch_threads(sys.argv[1])
tokens for testing :
https://www.mediafire.com/file/194pr1tulmh11mi/auth_tokens.txt/file

Hey @Unayung  @ilonabehn2
You can use this beta build for the time being.
I will release a stable version soon.


Extending Styled Components seems doesn't work in beta releasing.

I'm moving to use styled component and everything goes well except that extending style doesn't work at all,
I'm trying to rewrite my styled component ( Button ) by using
const NewButton = styled(Button)  // some new style
but nothing is applied on my style ?
Is there any suggestion ?

@mustafa-alfar I'm having the same issue and I'm getting a client-server mismatch warning for the classNames

Really can't help you at all if you don't use the issue template and provide some sort of reproduction case

@probablyup https://codesandbox.io/embed/hello-world-f364c

I figured out the problem,
I've wrapped the styled component by another component.
////////////
import styled from "styled-components";
import colors from "../../styles/colors";
const Btn = styled.button`
    font-size: 14px;
    font-weight: 100;
    border: 0.8px solid ${colors.blue};
    border-radius: 4px;
    min-height: 42px;
    display: flex;
    justify-content: center;
    align-items: center;
    cursor: pointer;
    outline: none;
    flex-basis: ${props => props.basis};
    color: ${props => (props.blue ? colors.white : colors.blue)};
    background-color: ${props => (props.blue ? colors.blue : colors.white)};
    &:hover {
        opacity: 0.9;
    }
`;
const Button = ({ children, blue, basis }) => (
    <Btn blue={blue} basis={basis}>
        {children}
    </Btn>
);

export default Button;

so I convert the code to >>
export default styled.button`
    font-size: 14px;
    font-weight: 100;
    border: 0.8px solid ${colors.blue};
    border-radius: 4px;
    min-height: 42px;
    display: flex;
    justify-content: center;
    align-items: center;
    cursor: pointer;
    outline: none;
    flex-basis: ${props => props.basis};
    color: ${props => (props.blue ? colors.white : colors.blue)};
    background-color: ${props => (props.blue ? colors.blue : colors.white)};
    &:hover {
        opacity: 0.9;
    }
`;

and that work fine.
Thank you