"token-account-history" error for the faucet

Running latest develop (commit ca5973299bbffc88f7531848811c26912a837f14)
Init network:
go run ./cmd/accumulated init -n "Badlands" --relay-to "Badlands"

Run node:
go run ./cmd/accumulated run -n 0

API request:
{
"jsonrpc": "2.0",
"id": 4,
"method": "token-account-history",
"params": {
"url": "acc://7117c50f04f1254d56b704dc05298912deeb25dbc1d26ef6/ACME",
"start": 0,
"limit": 10
}
}

API response:
{
"jsonrpc": "2.0",
"error": {
"code": -32802,
"message": "Validation Error",
"data": "jsonrpc2.Error{Code:ErrorCode{-32804}, Message:\"Accumulate Error\", Data:\"invalid pending object entry error unmarshaling GenTransaction signature length out of range\"}"
},
"id": 4
}

Attached ~/.accumulate as zip archive
Node0.zip
e.

This error has not recurred and running the commands above does not reproduce it on the current develop (bf14bc46506fc8ab576c5707c01ea41025eed896) so I'm closing it


Clarify the role of wildcards

Hello Team,
APIM support SSL certificate for custom domain with wildcard server certificates. For some case, (mostly cost saving of certificates) customer would like to treat specific domain differently while it can be contained in wild card domain.
For APIM, a specific subdomain certificate (www.example.com) would take precedence over a wildcard certificate.
Could you add this content to our doc ??
Besides changing certificates for specific domains, here is another possible scenario.  Assuming customer has 2 custom domains (private.example.com and *.example.com). They would like to protect only private.example.com with client certificate based on this doc . APIM also can negotiate client certificate only for private.example.com.
Similar behavior is explained in Cloudflare's product in detail, just for instance.
https://developers.cloudflare.com/ssl/reference/certificate-and-hostname-priority/

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: d02a5991-da24-06a2-0c24-d012d2e07399
Version Independent ID: 64acd893-43bb-81af-d14d-0b7553e93daf
Content: Configure custom domain name for Azure API Management instance - Azure API Management
Content Source: articles/api-management/configure-custom-domain.md
Service: api-management
GitHub Login: @dlepow
Microsoft Alias: danlep

@rui-hi
Thanks for your feedback! We will investigate and update as appropriate.

Thank you @Naveenommi-MSFT ! Sorry but I forget to write the main point to enhance. Customers concern the priority of wildcard domains. It should be the same as how wildcards works in DNS https://en.wikipedia.org/wiki/Wildcard_DNS_record.

@rui-hi - Thanks for this submission. I'm updating the article with this information. When merged, this PR will close this issue.


Make design system for Product Cards

Make and awesome design system for Product Cards Component.
See the Product Cards and make one.
Refer to : Button Design System for any reference
Tip :
You can use Tailwind Playground

can i work on this?

Assign this to me Already created solution for this issue


Shore up optional types, allow functions emulator config, and more

Checklist

Issue number for this PR: #nnn (required)
Docs included?: (yes/no; required for all API/functional changes)
Test units included?: (yes/no; required)
In a clean directory, yarn install, yarn test run successfully? (yes/no; required)

Description

Bump up to Firebase 5.5.7, which exposes Functions emulator configuration calls
Fix the types on @Optional DI. I had undefined which is incorrect, if a value isn't injected it is null. This was the "foot gun" that got me on the Functions region configuration.
Point the website to fboss
Switch the isPlatformBrowsers for Firestore and Messaging to !isPlatformServer allowing for use in web workers, etc.
Allow the Functions emulator to be used for callable functions via the FunctionsEmulatorOrigin DI token

Cleaning this up, writing docs, bump to 5.2, and will generate the changelog next.

Code sample

This is becoming too big IMO, going to split into multiple PRs


(docs): kitty usage

Add instructions on how to use tinty with kitty terminal

After mucking around with kitty for some time, I'm no longer interested in dealing with it and will just stick with Alacritty


emoji: Use parseUnicodeEmojiCode to render unicode emoji.

Messages contain emoji code with them in reactions. So use them instead
of relying on codes hardcoded in the app. This fixes the issue when
emoji names in app are outdated.
Fixes: #3716

I've also just merged two of the cleanup commits from this branch:
4738ec303 emoji [nfc]: Fix a parameter name on getFilteredEmojiNames.
8ee86a7be emoji [nfc]: Remove outdated TODO comment on Emoji component.

have getFilteredEmojis start returning type and code as well as name

commits are rearranged now.
c4bb2e73 (HEAD -> issue-3716, origin/issue-3716) emoji: Use emoji type to choose either from image or unicode emoji.
93e9d05e getFilteredEmojis: Return code as well.
1c98d3d7 getFilteredEmojis: Return emoji type.
73aa1ebe [nfc]test, flow: Add strict-local to emoji/data-test.

And merged! Thanks again @jainkuniya for your work on this.


Null function not behaving correctly

Hello All,
So I'm trying to use the null function to check the non-existence of a json entity in my data , but it is not behaving correctly:
My data is as follows:
[{"id":"0","labels":["Student"],"properties":{"id":20}},{"type":"stay","id":"1","labels":["Sport"],"properties":{"sport_id":100,"sport_name":"Tennis"}},{"type":"node","id":"2","labels":["Student"],"properties":{"name":"Venus Williams","id":10,"sport":100}}]
and my mappings are the following:
prefixes:
rr: http://www.w3.org/ns/r2rml#
foaf: http://xmlns.com/foaf/0.1/
xsd: http://www.w3.org/2001/XMLSchema#
rdfs: http://www.w3.org/2000/01/rdf-schema#
Airport: http://example.org/Airport/
ontology: http://data.sfgov.org/ontology/
grel: http://users.ugent.be/~bjdmeest/function/grel.ttl#
exfunc: http://example.com/idlab/function/
ex: http://www.example.org/
idlab-fn: https://w3id.org/imec/idlab/function#`
mappings:
episode:
sources:
- [data.json~jsonpath, "$.[*]"]
s: ex:$(id)
condition:
function: idlab-fn:isNull
parameters:
- [grel:valueParameter, $(type)]
po:
- p: a
o: ex:Node
The output is:
<http://www.example.org/0> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://www.example.org/Node> . <http://www.example.org/1> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://www.example.org/Node> . <http://www.example.org/2> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://www.example.org/Node> .
while the expected output should return only the first triple
Also, is there any way to use the not null function as well in RML?
Best regards,
Ali.

Could your paste the RML mapping file and properly format the yarrrml code? hard to bugfix this.
For the not null function: you can nest the not function with the null function, for an example of a nesting in a condition, see https://github.com/RMLio/yarrrml-parser/blob/development/test/condition-on-mapping-nested/mapping.yarrrml

Dear Ben,
so I attached both the yaml and mappings file as txt files:
bug_yaml.txt
mappings.txt
Best regards,
Ali.

So I'm trying to use the null function to check the non-existence of a json entity in my data

I assume you don't want to get a result when the type key does not exist in the JSON, by combining not with isNull, you can check if the type key exists and is not NULL. If that's the case, this mapping works:
Input
[
   {
      "id":"0",
      "labels":[
         "Student"
      ],
      "properties":{
         "id":20
      }
   },
   {
      "type":"stay",
      "id":"1",
      "labels":[
         "Sport"
      ],
      "properties":{
         "sport_id":100,
         "sport_name":"Tennis"
      }
   },
   {
      "type":"node",
      "id":"2",
      "labels":[
         "Student"
      ],
      "properties":{
         "name":"Venus Williams",
         "id":10,
         "sport":100
      }
   }
]

YARRRML
prefixes:
 ex: "http://example.com/"
 grel: http://users.ugent.be/~bjdmeest/function/grel.ttl#
 idlab-fn: "https://w3id.org/imec/idlab/function#"

mappings:
  person:
    sources:
      - ['data.json~jsonpath', '$.[*]']
    subjects: ex:$(id)
    condition:
      function: grel:boolean_not
      parameters:
        - parameter: grel:bool_b
          value:
            function: idlab-fn:isNull
            parameters:
              - [idlab-fn:str, $(type)]

    po:
      - [a, ex:Node]

Output
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix ex: <http://example.com/> .
@prefix grel: <http://users.ugent.be/~bjdmeest/function/grel.ttl#> .
@prefix idlab-fn: <https://w3id.org/imec/idlab/function#> .

ex:1 rdf:type ex:Node .

ex:2 rdf:type ex:Node .

If you don't want the not, you can just use the isNull function. You mapping contained a wrong parameter, you need idlab-fn:str. You can find them here: https://rml.io/docs/rmlmapper/default-functions/


The prompt says "click the 'Next' button to move on" but the button says "Run".

Repro steps from scrach:
Start any question
Complete a task within that question
Prompt will have inconsistent message with button -- button should change, I think.

Yep. There should be no "Next" buttons (or any other button) to press to move on to the next prompt. When you are done with the whole exercise and its associated prompts, then there is a "Next" button.

@rabidbit -- just to check, this is fixed, right?

Fixed in #61 .


Crash on fetching children of related field

Given a database with two tables: card and image, related as:

a card has (belongs to) one image
each image has many children images

the ORM query selecting all cards along with its image's children fails in a weird way:
await db.card.select({ images: q => q.image.children })

SELECT COALESCE("images".r, '[]') "images"
FROM "card"
LEFT JOIN LATERAL
  (SELECT json_agg(row_to_json("t".*)) r
   FROM
     (SELECT *
      FROM "image" AS "children"
      WHERE EXISTS
          (SELECT 1
           FROM "image"
           WHERE "image"."id" = "children"."parent_id")
        AND "children"."parent_id" = "card"."id") AS "t") "images" ON TRUE

Error: operator does not exist: integer = uuid

Complete reproduction:
import { createBaseTable, orchidORM, testTransaction } from 'orchid-orm'

const BaseTable = createBaseTable()

class CardTable extends BaseTable {
  readonly table = 'card'

  columns = this.setColumns(t => ({
    id: t.uuid().primaryKey(),
    image_id: t.integer().foreignKey('image', 'id'),
  }))

  relations = {
    image: this.belongsTo(() => ImageTable, {
      required: true,
      columns: ['image_id'],
      references: ['id'],
    }),
  }
}

class ImageTable extends BaseTable {
  readonly table = 'image'

  columns = this.setColumns(t => ({
    id: t.serial().primaryKey(),
    parent_id: t.integer().nullable().foreignKey('image', 'id'),
  }))

  relations = {
    children: this.hasMany(() => ImageTable, {
      columns: ['id'],
      references: ['parent_id'],
    }),
  }
}

const db = orchidORM(
  {
    databaseURL: process.env.DATABASE_URL,
    log: true,
  },
  { card: CardTable, image: ImageTable },
)

await testTransaction.start(db)

await db.$query`
  create table "card" (
    id uuid not null primary key default gen_random_uuid(),
    image_id int not null
  );
  create table "image" (
    id serial not null primary key,
    parent_id int
  );
`

await db.card.select({
  images: q => q.image.children,
})

await testTransaction.close(db)

the ORM query selecting all cards along with its image's children fails in a weird way:

await db.card.select({ images: q => q.image.children })

ORM doesn't yet support selecting relation of relation like this, only a direct relation.
You can define a relation using through option, so that card has many children through image, and then it should work fine.

Understood, thanks!
Is traversing this kind of relations something that you have plans for? If not, can it be somehow disabled at typings level?
Currently, it makes an impression that it will work:
const cards = await db.card
  .select("id", {
    image_url: (q) => q.image.children.where({ size: "small" }).get("url"),
  })
// cards is typed as: Array<{ id: string, image_url: string }>
// which is what a developer would actually expect.

I didn't have plans for it until this issue, I think it would be nice to have.
But as it seems to be a big one, I'd like to postpone it to be done after one current big change I'm working on, perhaps for a few weeks.

For the reference, in this particular case, the through relation worked perfectly, thanks for the tip:
class CardTable extends BaseTable {
  relations = {
    image: this.belongsTo(() => ImageTable, {
      required: true,
      columns: ["image_id"],
      references: ["id"],
    }),
    image_children: this.hasMany(() => ImageTable, {
      through: "image",
      source: "children",
    }),
  }
}

const cards = await db.card
  .select("id", {
    image_url: (q) => q.image_children.where({ size: "pre-middle" }).get("url"),
  })

I think there is no point trying to support this syntax:
image_url: (q) => q.image.children

Because that means computing types of all relations of all relations just in case, TS won't survive this.
And if we try to compute relations of relations of relations, that would lead to endless recursion and TS will die even harder.
image_url: (q) => q.image.children.image.children.image.children,

It makes me wish there was an alternative language that could compute types lazily, on demand, without limits, and without heating my laptop.

I believe there was a (now deleted) comment which elaborated that there were no point trying to support this syntax:
image_url: (q) => q.image.children.image.children.image.children

due to Typescript limitations.
I am not competent enough to argue, but, my original point was that if that is not supported, perhaps it would be possible to somehow disable that kind of traversal it at typings level? Currently, this kind of query aligns with developer expectations from this great ORM, it types properly (or at least seemingly so in simple cases), it compiles, it runs, it even builds a syntactically correct SQL query. Only there the problem starts: the SQL query is logically incorrect (and possibly the result parsers won't work either).

I wrote a message and deleted it because after trying it seems to be possible, will have to only slightly override types of nested relations to always return multiple records, because when querying "has one" that's chained after "has many" it should return many.

The feature is ready!
In the latest release, it's possible to chain relations however deep in select, and each chained relation can have (optionally) its own where conditions and any other methods such as limit, offset.
// should work now:
await db.card.select({ images: q => q.image.children })

await db.card.select({
  images: q =>
    q.image.where({ imageColumn: 'xxx' })
      .children.where({ childColumn: 'yyy' })
      .limit(10)
})

Also, it's now supported to have chaining on a top-level, without select.
// Loads children with conditions
// `where exists` image with conditions
// `where exists` card with conditions

await db.card.where({ ...conditions })
  .image.where({ ...conditions })
  .children.where({ ...conditions })


Train use RGB image, but inference use BGR image?

Train network, use cv2.read function load image, and use trans.ToPILImage() function conver to RGB mode. but at the stage of inference, the image only use cv2.read function load image.

We have rewritten the ToPILImage in the transform.


Support for Softlayer multi-path feature

This is needed for Bluemix on SoftLayer (SL) since the best disks on SL are iSCSI and the changes are needed to correctly setup these disks. Most of this code was part of a forked copy of the agent code that the Bluemix team has been maintaining so it's already in production but we want to contribute this back.

Hi, all,
@gu-bin is part of the IBM organization and working with me here in Beijing. I will have her added to the cf-ibmers shortly. Best,
dr.max

new PR #29 was created to replace this one.


SPIKE: Research existing Monitoring Plan check engine

Dive into existing MP check engine and understand its capabilities

Emissions/QA out of scope, just looking into how check engine validates MP data
Next ticket: Design BRE system

Meet with ERG and received a spreadsheet of all the MP checks with details regarding categories and order of processing. Have look through existing check engine get familiar with the process and reviewed check specifications.
High Level Architecture
https://github.com/US-EPA-CAMD/diagrams
https://github.com/US-EPA-CAMD/diagrams/blob/develop/EPA CAMD BRE Check Engine.pdf
Built preliminary UI for Rule Authoring
https://easey-dev.app.cloud.gov/admin/rules
Built preliminary API for Rule Authoring
https://github.com/US-EPA-CAMD/easey-rules-api
Built preliminary Scheduling & Evaluation process
https://github.com/US-EPA-CAMD/easey-job-scheduler


Sorting dependencies in pyproject.toml does not work as expected

Related issues
Relates to #6203.
Setup

uv 0.5.11 (Homebrew 2024-12-20)
Python 3.13
macOS Sequoia 15.1.1

Observed behavior
When adding a series of dependencies with uv add:
uv add ruff
uv add pytest
uv add pytest-mock
uv add pytest-randomly

this is what I get in my pyproject.toml:
[project]
# ...
dependencies = [
    "pytest-mock>=3.14.0",
    "pytest>=8.3.4",
    "ruff>=0.8.4",
    "pytest-randomly>=3.16.0",
]

The list of dependencies is updated as follows:

uv add ruff: ruff is added.
uv add pytest: pytest is inserted before ruff.
uv add pytest-mock: pytest-mock is inserted before pytest.
uv add pytest-randomly: pytest-randomly is appended as the last item.

So, sorting is broken at point 3.
Expected behavior
Since uv should sort the dependencies alphabetically (see #6388), I would rather expect:
[project]
# ...
dependencies = [
    "pytest>=8.3.4",
    "pytest-mock>=3.14.0",
    "pytest-randomly>=3.16.0",
    "ruff>=0.8.4",
]

or, assuming that >= are taken into account in sorting:
[project]
# ...
dependencies = [
    "pytest-mock>=3.14.0",
    "pytest-randomly>=3.16.0",
    "pytest>=8.3.4",
    "ruff>=0.8.4",
]

I believe that the first options seems to be more intuitive. In fact, by "sorting" dependencies I would rather think of sorting dependency names, not its specifiers.

Thanks! Looks like a bug.


Dox

cc @EPashkin

I thought that possible problem with dox dependence on glib and it may work wrong without "use_glib".
But now I think that it not problem actually.
So 👍

Yeay!


Cleanup jobs not properly created for TaskChain + KeepOutput=False

More info in the egroup discussion, subject "CERN unmerged area investigation"
Pasting here the important information:
...
From Sid
I've put two lists in [1]. eos_dump.txt contains a directory breakdown of the eos unmerged space, as well as the total (1.45 PB). other_sites.txt contains the size of unmerged at other sites, according to the dynamo consistency tool. There are some caveats as to what goes into the latter (for example, RAW is always ignored), but it should be a good estimate.
[1] http://t3serv001.mit.edu/~snarayan/browse.php?dir=transfers/unmerged_dump
...
and my investigation
...
thanks for the eos dump!
I sorted those directories by data volume and there is around 40 directories with >= 10TB
of unmerged data. Then I picked the worst offender (41 TB):
/store/unmerged/RunIIFall17wmLHEGS/W1JetsToLNu_LHEWpT_150-250_TuneCP5_13TeV-amcnloFXFX-pythia8
and started debugging it from there. I made several notes and checks here and there and you
can see all of that in this text file:
https://www.dropbox.com/s/qvavbp9qv41jprn/debug_store_unmerged_CERN_EOS.txt?dl=0
The workflow creating those output datasets was acquired by 4 different agents (plus a bunch
of ACDC workflows), but I did all this investigation in one agent only - which is fine.
In short, I think it's safe to acknowledge there is a bug in WMAgent regarding Cleanup job
creation for TaskChain workflows with KeepOutput=False (staging the unmerged output but
not merging neither injecting into DBS/PhEDEx). This seems to affect only the specific output
that is used as input for further processing tasks though (see LHE check on the text doc).
Note, however that the workflow I used for this investigation was announced yesterday, so it
shall be archived today and all the agent information gone (that also means that unmerged
path will get unlocked by Unified and cleaned up by Dynamo soon, I think).
I'm creating a WMCore issue with the content of this email, but that shouldn't stop us from
safely releasing /store/unmerged space in a different way. I bet this cleanup job creation
has a nasty logic and it's not going to be straight forward to get it fixed.
PS.: Another thing that I cannot explain is why around 15k unmerged files were not processed
by a further Processing job (186478 out of 203950 files)!?!
...

@ticoann before we start working to fix it, can you please go through my text file and double check my steps, especially the sql queries just to make sure I didn't make any mistake and we really have this issue in the agent code.

Just to make it harder to debug... it doesn't seem to happen to all such requests. Looking at this workflow (already in completed)
prozober_task_HIG-RunIIFall17wmLHEGS-00746__v1_T_180714_121808_9488
with Task1 and Task2 KeepOutput=False, I cannot find any unmerged leftovers in EOS, even though looking at vocms0257 database it tells me not all unmerged files have been assigned to cleanup jobs.
There are 74 unmerged GEN-SIM-RAW files created:
select count(distinct lfn) from wmbs_file_details where lfn like '/store/unmerged/RunIIFall17DRPremix/GluGluToHiggs0PHf05ph0continToZZTo2e2mu_M125_10GaSM_13TeV_MCFM701_pythia8/GEN-SIM-RAW/PU2017_94X_mc2017_realistic_v11-v2/%';
COUNT(DISTINCTLFN)
------------------
		74

while only 70 of them have been assigned to a Cleanup job (74 assigned to Processing jobs)
SELECT wmbs_sub_types.name, count(DISTINCT wmbs_job.id), count(DISTINCT wmbs_file_details.id)
  FROM wmbs_file_details
    INNER JOIN wmbs_job_assoc ON wmbs_job_assoc.fileid = wmbs_file_details.id
    INNER JOIN wmbs_job ON wmbs_job.id = wmbs_job_assoc.job
    INNER JOIN wmbs_jobgroup ON wmbs_jobgroup.id = wmbs_job.jobgroup
    INNER JOIN wmbs_subscription ON wmbs_subscription.id = wmbs_jobgroup.subscription
    INNER JOIN wmbs_sub_types ON wmbs_sub_types.id = wmbs_subscription.subtype
    INNER JOIN wmbs_workflow ON wmbs_workflow.id = wmbs_subscription.workflow
    INNER JOIN wmbs_location_pnns ON wmbs_location_pnns.location = wmbs_job.location
    INNER JOIN wmbs_pnns ON wmbs_pnns.id = wmbs_location_pnns.pnn
  WHERE
    wmbs_file_details.lfn like '/store/unmerged/RunIIFall17DRPremix/GluGluToHiggs0PHf05ph0continToZZTo2e2mu_M125_10GaSM_13TeV_MCFM701_pythia8/GEN-SIM-RAW/PU2017_94X_mc2017_realistic_v11-v2/%'
    GROUP BY wmbs_sub_types.name;
NAME			COUNT(DISTINCTWMBS_JOB.ID) COUNT(DISTINCTWMBS_FILE_DETAILS.ID)
--------------- -------------------------- -----------------------------------
Cleanup 		31						    70
Processing		55						    74

BTW, we have two templates exercising that same TaskChain setup:
WMCore/test/data/ReqMgr/requests/Integration/TaskChain_MC.json
WMCore/test/data/ReqMgr/requests/Integration/TaskChain_DropOutputs.json
and none of them have uncleaned unmerged files.

@amaltaro @ticoann can you please update this and come up with something quite soon about this

Hi Jean-Roch, we are still investigating. Alan found out that files not which is not cleaned are actually used for the failed jobs. (checked prozober_task_HIG-RunIIFall17wmLHEGS-00746__v1_T_180714_121808_9488). Which is expected since we can't clean up unmerged input files which is failed unless ACDC is created and succeeded which was original design.  We are trying to check some other workflows whether that is consistent and actually that is the sole cause of unmerged files not deleted. If it is the case, I don't think that is the bug. We have have some other way to clean up those files fast. (Dynamo delete those files eventually with the lock condition, correct?)

Another suggestion -
Can we tally up the breakdown of unmerged size at CERN by workflow priority?  That would allow a different kind of sanity check.
If we see that 80% of unmerged space goes to low-priority workflows -- well, there's not much we can do other than change how we interpret priority.

@ticoann @amaltaro is there a way to list all unmerged files that are still in use from the agent, so that we dump this somewhere for dynamo to use and clean files that were not cleaned by the agent in the first place, but that would live in a directory that is still protected

@bbockelm, Here is the list of unmerged area sorted by Alan's document above.
request and priority related to that.
The structure is
umerged area  size (TB)
     request name,  request status,  initial priority, request priority

https://www.dropbox.com/s/2k1hnq505zzp35y/umerged.txt?dl=0
The script is just checking whether certain workflow is contributing data in given unmerged space.
So it doesn't mean it is responsible for all the space. It might not contribute at all. It also only includes the current active requests (not archived) so it might be possible actual contribution of data is from other workflows which already archived (and space might be cleaned up).
So it is just a rough view what kind of priority might contribute on above unmerged space.
That said, I am not sure what is considered low priorities. Comparing current active workflows, I think most of them are below the median  among currently running workflows.

is there a way to list all unmerged files that are still in use from the agent

I think we can get the unmerged acdc inputs from db. But I am not sure I follow the logic about cleaning. You want to move those files on specific location so dynamo can deleted later?
Sorry if I misunderstood.

In terms of priorities - let's call anything less than 100k at CERN "low priority".
Can we make this simpler by only looking at files WMAgent knows about?  We can separate out files that exist but not known to the agent (although it'd be an interesting sub-thread to attempt to count these!).

One other thing is we still not sure whether Agent has a bug or not. I will try to test at least few more workflows which doesn't create the clean up jobs falls into failed job case.

Let me jump into this discussion. I think we're trying to solve it the hard way, to be frank.
Fetching unmerged files known by the agent that isn't anymore needed by any job is a total overkill and waste of time in my opinion. Keep in mind that we don't run a single agent and there is not a single database for fetching this info (besides the ACDCServer, but that would be terribly expensive).
My suggestion still stands, why don't we reduce the waiting period of unlocked LFNBases in dynamo and let it clean files older than X days (why not 1 day?) instead of the current 28 days. I think Dan mentioned there would be a few hundred TBs that would be cleaned right away. It doesn't need to be anything permanent nor anything for all sites, just to CERN such that we can start using those 70k resources sooner than later.
PS.: Honestly, I also fail to see what role the workflow priority is playing here. Are you considering aborting/cleaning up and running that (low) priority workflow elsewhere?

That's why I specifically only was interested in looking at one agent - I didn't want to try to construct a global picture.
The role the workflow priority plays here is to help understand why there may be so many files sitting around.  Lots of unmerged files from low-priority workflows are "understood" more than unmerged files from high-priority files.  Are the majority of the files "understood" or "not understood"?
Basically, I'm looking for two numbers -- for agent XYZ, the # of TBs of unmerged data associated with CERN above prio 100k and the number below 100k.
I'm not thinking about any concrete action with these yet, just trying to quantify the different causes.
I agree that understanding unlocked LFNs are important.  We should probably file a separate ticket for that because they are important but a different topic.

I just looked at the list that @ticoann provided, and it struck me that the first workflow that is taking 41 TB is not even running
https://cmsweb.cern.ch/reqmgr2/fetch?rid=pdmvserv_task_HIG-RunIIFall17wmLHEGS-01778__v1_T_180521_202854_6570
Is this possible? I thought the unmerged area is only used for the intermediate unmerged output datasets?

Is this possible? I thought the unmerged area is only used for the intermediate unmerged output datasets?

Hi Thong, That is what I was explaining above.

The script is just checking whether certain workflow is contributing data in given unmerged space.
So it doesn't mean it is responsible for all the space. It might not contribute at all. It also only includes the current active requests (not archived) so it might be possible actual contribution of data is from other workflows which already archived (and space might be cleaned up).

For the particular space, workflow responsible is already archived. pdmvserv_task_HIG-RunIIFall17wmLHEGS-01115__v1_T_180415_203803_4286

OK I take that single workflow does not take the whole 41 TB, but how come it contributes any data to the unmerged space given that it's not running?
I also agree with Alan's suggestion of reducing the waiting period for unlocked to 1 day -- from the list I see that most of the top spaces are having only 0 to 2 active workflows, so the majority of the unmerged space comes from the unlocked data of archived workflows that are waiting to be deleted.

https://dmytro.web.cern.ch/dmytro/cmsprodmon/workflows.php?prep_id=task_HIG-RunIIFall17wmLHEGS-01115 has created files that the presence of https://cmsweb.cern.ch/reqmgr2/fetch?rid=pdmvserv_task_HIG-RunIIFall17wmLHEGS-01778__v1_T_180521_202854_6570 locks for dynamo

Can I suggest we wait one day more to complete this investigation. Seangchan and I are almost sure we found the reason for these leftovers, we "just" need to do some extra checks before coming up with that solution. Part of the conclusion is on Seangchan's comment:
https://github.com/dmwm/WMCore/issues/8723#issuecomment-411125499
and if that's really true, it's not really a bug but a feature of ACDCs :-) Workflow priority has no effect as well, it's all a matter of "a failed processing job that was reading an unmerged file", thus this unmerged file is put in the acdcserver and we take that file out of any cleanup jobs (such that an ACDC can run on it).

hey @amaltaro , if (and only if) there is a bug somewhere that left >1.4PB of unmerged file at CERN (all other places at T1/T2 will run into critical situation at some point) the heavy query on wmbs/agent to get an actual list of protected files, and take evasive action on eos (remove all old files not explicitly protected by that list) is something we have to do to mitigate the issue.
as for lowering down to 1 day, it might be too aggressive (if the list does not update often enough), but 28 is way too much ; has anyone contacted directly transfer team to make this change?

Hi all,
I'm still really uncomfortable sweeping a petabyte under the rug, shrugging our shoulders, and saying "WMAgent leaks ten terabytes a day, but Dynamo will clean it up".
We have identified one or two possible causes.  Can we estimate what percentage of the files on disk these account for?
If we are chasing files that can be processed by a future ACDC - can we create a monitoring view of this metric?
Brian

After changing this message a few times before posting it, these were my findings while analyzing another sample workflow:
pdmvserv_task_HIG-RunIIFall17wmLHEGS-01116__v1_T_180418_043017_9486
somehow what's reported by WMStats does not match what is in the ACDC Server. I believe because we had a redeployment campaign (1.1.12) between April and May and that data got lost.
I guess that's expected, but if Seangchan can confirm(?) If it's not expected, we might follow that up in another issue not to spam too much here.
Ok, then adapting and running an old script from 2015
https://github.com/amaltaro/scripts/blob/master/checkACDCCollections.py
and querying the ACDC server for this workflow above, I confirm that CERN EOS has almost the same number of unmerged files in its storage. Check the output of that script against a few eoscms queries:
http://amaltaro.web.cern.ch/amaltaro/forWMCore/Issue_8723/pdmvserv_task_HIG-RunIIFall17wmLHEGS-01116__v1_T_180418_043017_9486
so the only file leak (well, at least that I cannot explain now) happened on the LHE datatier, around 100 files left in the storage, instead of only 3.

I'm still really uncomfortable sweeping a petabyte under the rug, shrugging our shoulders, and saying "WMAgent leaks ten terabytes a day, but Dynamo will clean it up".

Thus, I'd look at it from another angle. When ACDC was designed it was counting on the site cleanup policy to clean any files meant to be processed by ACDC workflows, which were not really ACDC'ed.
It was a flaw in the design and now we need to come up with a solution for such cases, which I still don't have and likely won't be simple.

as for lowering down to 1 day, it might be too aggressive (if the list does not update often enough), but 28 is way too much ; has anyone contacted directly transfer team to make this change?

Call it a week then. Dan said the cleanup script runs only once a week anyways, so if it's a day it will be cleaning data older than 1 to 8 days, if a week data older than 8 to 15 days.
What is the actual concern with this file age here? Are there any cases where a workflow is announced and requestors come back asking for the final lumis to be recovered (if so, how often?)?
Otherwise, how the list is built looks pretty safe to me:
https://github.com/dmwm/WMCore/blob/6c0c1283374b25a9d0d9a3ffbc59af3777c29a13/src/python/WMCore/ReqMgr/DataStructs/RequestStatus.py#L121

Thanks Alan for the investigation.

somehow what's reported by WMStats does not match what is in the ACDC Server. I believe because we had a redeployment campaign (1.1.12) between April and May and that data got lost.

What is not matching? If the number of failed jobs, yes, it is possible between the redeployment. But it shouldn't be many job report loss unless there was replication problem.  We are redeploying agents after all the jobs are finished and upload to DBS and PhEDEx. Although we don't check replication status before redeploy the agent, it all the reports should be reported by then.  Other possibility is we just lost the report update in local couch db. When failed job report comes back we try to update once and if it fails we don't retry. (That was by design with the assumption that local couch is reliable and we can afford to lose some reports for monitoring).

What is the actual concern with this file age here? Are there any cases where a workflow is announced and requestors come back asking for the final lumis to be recovered (if so, how often?)?

Just a little more detail about what Alan said. As far as I understood when workflow is announced by unified, it change the status of all the related workflow (ACDC, cloned, recovery) to announced.
Then the umerged lfn base is remove the protected list. I haven't even thought about the use cases what Alan mentioned above. It the concern is whether protected lfn list is constructed correctly as intended we should double check and make sure.

@amaltaro "What is the actual concern with this file age here?" just making sure the file is not newer than when the list of protected directory was updated last : a week would do.

@amaltaro  regarding https://github.com/dmwm/WMCore/blob/6c0c1283374b25a9d0d9a3ffbc59af3777c29a13/src/python/WMCore/ReqMgr/DataStructs/RequestStatus.py#L121 could have assigned and acquired removed from the list of statuses to lock on, since those guys have not yet created anything by any mean (https://github.com/CMSCompOps/WmAgentScripts/commit/8f8772362a8b4e062f9b65127ed0320a8d2a11b7)

concerning the issue, there has been some back and forth and I want a clean and definite answer @amaltaro @ticoann on whether files are left behind from task of taskchain with KeepOutput=False.
Is this a "yes files are left behind" (bug in wmagent), or "no, files are properly cleaned up, if not the source of a failed job" (no bug, and those files are required). You can take a couple of days as per https://github.com/dmwm/WMCore/issues/8723#issuecomment-411227958 request

could have assigned and acquired removed from the list of statuses to lock on, since those guys have not yet created anything by any mean (CMSCompOps/WmAgentScripts@8f87723)

There is always race condition for when status is queried and and actual deletion happens (status change  in mean time). However, I think it is safe if we wait at least 1 day before deletion.
I was wondering whether it cause more problem including those? If that is the problem having more list,  we can remove those status. ('assigned', 'acquired')

@amaltaro - this looks like great progress!
To make sure I understand:

These are files known to the ACDC server
We've decided (for whatever reason, doesn't matter) to not run an ACDC for this workflow.
ACDC is never informed of this fact.  Even if it was, there's no ACDC "cleanup" mechanism that it can kick off.

Does the system know when we won't actually run the ACDC?  I always feel better stating the files it is safe to delete as opposed to locking files.

Does the system know when we won't actually run the ACDC? I always feel better stating the files it is safe to delete as opposed to locking files.

When original workflow is announced which means data ops are ok with  data produced already and unified announce all the related ACDCs (@vlimant Jean-Roch can confirm). Than we are safe to delete the files except for some reason people decided to collect more data from already announce workflow. (As Alan was mentioning). However ACDC data will be cleaned up (not the actual files but metadata from couch db), once workflow is archived (There is a grace period) so it is hard to even produce new data.
We provide REST api for protected lfn base: (means the unmerged files which might be still in use)
https://cmsweb.cern.ch/wmstatsserver/data/protectedlfns
Once a workflow is announced the unmerged lfn base (for output data) from the workflow will be removed from above API result.

Trying my best to give a clean reply, since we've a few sort of diff topics on going now :)
About producing the protectedlfns list, I think it doesn't hurt us much if we keep those two status (assigned, acquired). That's also a safeguard in case the request transition thread goes down or doesn't run for some unexpected time.
About the cleanup policy, I suggest we follow that up in another thread. One week should do the job for now, probably not in a year or two though. We might also push for the new policy not only for CERN, but everywhere. Anyhow..
@bbockelm yes, you summarize well the issue. As Seangchan mentioned, IF ACDC is not created, then we rely on the sites (now DM) to clean those files. Once the workflow gets archived (or announced) in reqmgr2 - I just checked the code, apparently there is no grace period - all the ACDC information is cleaned up.
Jean-Roch, I'm going to perform the same check on a couple of other requests, but I believe (hopefully) the conclusion won't change.

Ok, I've done ACDC x EOS comparisons for 4 more requests in completed (but with ACDCs already running):
pdmvserv_task_SMP-RunIISummer15wmLHEGS-00227__v1_T_180416_194004_2476
prozober_task_SMP-RunIISummer15wmLHEGS-00226__v1_T_180611_192517_1020
pdmvserv_task_HIG-RunIIFall17wmLHEGS-01122__v1_T_180415_203643_8440
pdmvserv_task_HIG-RunIIFall17wmLHEGS-01114__v1_T_180415_203757_5745
and in general the number of unmerged files in the ACDCServer is very close to the amount of files under CERN EOS paths. If you want, you can see my notes and outputs in the dropbox file I shared in my first comment.
I used this script for the EOS checks:
https://github.com/amaltaro/scripts/blob/master/checkEOSPath.py
In my opinion, WMAgent is not leaking files and those leftovers belong to jobs that failed, so the agent doesn't delete because we might run ACDC on them. As already explained above.
PS.: the only case I see more files than expected is for the LHE tier. However, it could be that there were cleanup jobs failing (wmstats got rotated, so we can't be sure how many failed for these old workflows). To be investigated again in the near future...

thanks

closing the issue since it seems agent doing what it is supposed to do.


Update support-version.list for Pulsar test scenario

Replace 2.7.0 with 2.7.3

Sorry, looks like 2.7.3 have not synced to maven central yet, will update after the package is available in maven central.

Sorry, looks like 2.7.3 have not synced to maven central yet, will update after the package is available in maven central.

Is the release process done?

@wu-sheng Yes, released but waiting for sync to the maven central.

@wu-sheng Yes, released but waiting for sync to the maven central.

Usually this sync doesn't need so long time, please check whether release manager has clicked the release from ASF nexus to maven central

@kezhenxu94 @wu-sheng I have removed 2.8.1 due to there are some change makes the plugins does not compatible with 2.8 anymore, I will follow the 2.8.x compatibility issue. For the 2.7.3, it is available in Maven Central for now


'NoneType' object has no attribute 'op'

I am trying to run the code for my model which uses 3d convolution and fully connected layers.
grads = gradient_memory(train_loss, self.model_variables)
grads = list(zip(grads, self.model_variables))
This should give me the list as
optimizer.compute_grads(train_loss, var_list=self.model_variables)
But instead, I get:
File "gradient_checkpointing.py", line 274, in
inputs_to_do_before = [d_checkpoints[r].op for r in ts]
'NoneType' object has no attribute 'op'
Can you help me with this, please?
I have set the checkpoints equal to ts_all.

I got the same error when one of the tensors in checkpoints did not have any gradient (also implicit via checkpoints='speed' or so).
Removing those from checkpoints fixed the issue for me.


[SPARK-41757][CONNECT] Fixing String representation for Column class

What changes were proposed in this pull request?
To reenable the doc tests for Column, this patch fixes the string representation of the Column, in particular this introduces a workaround for a binary unresolved function that produces an infix representation of the function string.
Why are the changes needed?
Compatibility
Does this PR introduce any user-facing change?
No
How was this patch tested?
Reenabled doc tests.

Can one of the admins verify this patch?


Adding karma-coverage

Hello, Ben.
I was tinkering with karma-coverage to see if I could get it working, and I haven't succeeded. I'm getting an unexpected token < error from framework.browserify when I add browserify-istanbul (or istanbulify) as a transform. Google leads me to believe that this error is universally understood to mean "something really obscure is broken, and we don't know what it is. Good luck figuring it out."
It needs something like this in karma.conf.js:
browserify: {
	debug: true,
	transform: ['debowerify', 'browserify-istanbul'],
	extensions: ['.js']
},
reporters: ['spec', 'coverage'],
coverageReporter: {
      type : 'html',
      dir : 'coverage/'
},

The docs of karma-coverage suggest using 'coverage' as a preprocessor, but we can't do that because of Browserify. Apparently browserify-istanbul or istanbulify can be used to get around that, but, as I said, I haven't succeeded.
What do you think about karma-coverage? Have you already set up such a tool with this architecture before? Do you maybe prefer a different tool? I'd like to hear your opinion.
Links:

karma-coverage
browserify-istanbul
istanbulify
Some guy apparently got something similar working here
... and another one here

Hey man, I managed to get it working with a few simple changes. Have a look at the develop branch and see if that does what you need it to. I admit I don't have much experience with code coverage tools, so I don't know if it's doing what it's supposed to, but give my changes a try and let me know if further changes are needed.
Cheers!

I've personally never used a code coverage tool. Honestly, I've barely used unit testing, but I think it's important to figure it out. Those *.spec.js files were calling my name last night, but I wasn't really sure where to start. I'm hoping that something like karma-coverage can help me learn.
I can't test it until I get home this evening, but two things are on my mind after looking at your commit:

You switched 'build-dev' to 'build' in the 'build-test' Gulp task. This means that the 'vendorJS' task also gets run, though, so that's going to slow it down. Maybe you did this because you couldn't guarantee when running gulp test that the vendor.js file was available in ./dist/? Maybe there should be a 'build-test' task which uses 'build' as well as a 'build-test-dev' task which uses 'build-dev'?
I see that you sent !(*spec)*.js to the 'coverage' preprocessor. I was trying to use browserify-istanbul or istanbulify transforms to instrument the files in Istanbul because Google led me to believe that Browserify needed to stitch together all of the files into one bundle.js file before Istanbul could work. Apparently that's not the case.

I changed "build-dev" to "build" to quickly and easily wipe the dist directory (which contains the coverage reports). It was quick and dirty so that you could take a look. I definitely think that a build-test task should be implemented though.
This all being said, I'm not sure how to et the coverage to work as I'd expect it to. It shows 100% coverage for everything, when I tried adding a new function it didn't seem to detect that it had no coverage.
I definitely want to get this working, though. Like you, I agree that this is important for testing. Most of my teatig experience is with integration and functional testing as opposed to unit testing. Unit tests, however, are something I definitely want to learn myself. I'll check with my colleagues at work on Monday and see if they have any insight into this.

Well, I've made a little bit a progress by including Partialify. It got me past the unexpected < token error. I'm getting an HTML report which includes the test coverage data for everything in /libs/ with this code, and I'm still not getting any meaningful report for anything in /app/. It seems to me that it isn't actually pulling in the JavaScript files and looking at them, but I'm not really sure what to do about that. It just has one line for each file, claiming 100% coverage of the line. Each one looks like this:
Statements: 100% (1 / 1)      Branches: 100% (0 / 0)      Functions: 100% (0 / 0)      Lines: 100% (1 / 1)      Ignored: none
require("c:\\Users\\Brandon\\Documents\\GitHub\\Optical\\app\\common\\constants\\CONSTANTS.js");

Here's my karma.conf.js file as it is now:
var istanbul = require('browserify-istanbul');

// Snip

preprocessors: {
	// './dist/bundle.js': ['browserify'],
	'./app/**/*.js': ['browserify', 'coverage']
},

browserify: {
	debug: true,
	transform: [
		'partialify',
		'debowerify',
		istanbul({
			'ignore': ['**/node_modules/**', '**/*.spec.js']
		})
	],
	extensions: ['.js']
},

reporters: ['spec', 'coverage'],

So, if I add '**/libs/**' as something that Istanbul ignores, then that solves the problem of writing a coverage report for the libraries. Of course, we don't care about the coverage report of the libraries, but I figured I'd mention that karma-coverage works at least for that.
However, when it generates the report for '**/libs/**', it also has an error. TypeError: Cannot read property 'split' of undefined. You may have noticed that this error was also faced by someone in the thread that I linked. It also may be worth pointing out that one of the karma-coverage contributors claims that the problems described in that thread should have been fixed in a release from 11 days ago...
Links:

angular-node
A similar project that apparently got this working
gulp-istanbul
gulp-karma

Removing 'coverage' from the preprocessors resolved the cannot read property 'split' error. I'm making progress on this very... slowly... I'm now able to get complete, functioning reports for the libraries. Also, when I removed the preprocessor, it completely lost track of anything in /app/, which in my opinion confirms that browserify-istanbul isn't picking them up.

Not sure what to tell you just yet. My changes got the coverage report generating, but I'd get 100% in all fields, and if I added a function somewhere, it didn't seem to detect it. I didn't get a chance to come back to this over the weekend. You've gone into this much more heavily than I have.

I'm also not sure why partialify would have helped with this, as the current starter kit doesn't use partials in the traditional manner.

The snippet that I posted yesterday gets a proper coverage report. It keeps track of if/else branches, function coverage, and so on. The problem is that it's redoing the bundle, and that's a waste in my opinion. I'd like to somehow include Istanbul and Karma in the existing build-dev task. I tried yesterday, but I failed.
I can't say for certain why Partialify is needed because I honestly don't understand everything. It's something that I saw used by someone else who faced a unexpected token < error, so I tried it, and it resolved that error. I don't understand why we don't need Partialify because we are in fact requiring HTML files.

Ultimately, the coverage report should only be generated while testing. It shouldn't be created while running the dev tasks.

Partialify is probably "needed" because browserify does need to bundle HTML files somehow. In the case of this starter kit, we're using the transform "html2js" which converts the HTML to JS files so that they can be browserified.

I'll try and look at my implementation more tonight and see what I can come up with.

"Only generated while testing." Are you sure? So, you think that I should run gulp test to run my tests and get a coverage report, and that should be it? The gulp stream would end. No autowatch. No rebundle. If that's the case, then what I've written can do that. Here's what it does:

Run gulp karma
The karma-browserify 'browserify' preprocessor accepts app.js as the entry point and bundles the files.
The Partialify transform lets karma-browserify bundle up the HTML files.
The browserify-istanbul 'istanbul()' transform instruments the files in the bundle, but it doesn't instrument *.spec.js files or files in /libs/.
The Istanbul report is written by the karma-coverage 'coverage' reporter. It is a complete and proper HTML Istanbul report.

Even if a separate test/coverage task is the desired outcome, I still think that the browserify/istanbul aspect of it should be brought into the gulpfile instead of being written in karma.conf.js. Specifically, I want to use the existing vendor.js bundle to speed up the karma-browserify part. I think that working with gulp-istanbul and gulp-karma.
However, I was looking at the autowatch feature of Karma, thinking that the workflow might be something like this:

Run gulp
Code gets bundled.
Tests get run.
Coverage report is generated.
Change some code. Save the file.
Steps 2 through 5.

On second thought, that would definitely cause a drop in the performance of the rebuilding even if I did use an existing vendor.js bundle. I'm also not sure if there's a real value to re-running tests after saving the file. Maybe not. What do you think about that?

So the reason that the coverage report should be tied to the "test" task is because the coverage report is only relevant to Karma (which is only run during the "gulp test" task). The dev task doesn't run any tests, so coverage has to be tied to the "test" task.
Typically, coverage reports are generated during Continuous Integration builds that run all of the tests and generate the reports after the fact (CI could be Travis, Jenkins, Team City, etc).
I've currently got things set up so that when the "test" task is running, the tests automatically update when code is changed (saved). That tends to be the way to do things so that as you're writing your code, you can quickly and easily see if your tests pass or not.
It's looking like I may not get around to accomplishing much tonight. I've been trying to get browserify-istanbul installed, but that package seems to take an eternity to install. It's ridiculous.

Got it working. However I can't seem to figure out how to exclude the libs directory from the coverage report. Maybe you can take a crack at it. I've pushed my changes to the develop branch.

Excellent work fixing the last part of this. Well done!

It took me over an hour to write this. I got caught up reading about various things. ^.^"
I created pull request #23. If you decide that you want to use karma and istanbul, then this matter is closed for you. However, it's the wrong way to do it in my opinion.
Here's the reason:

gulp test is executed
'build-dev' is executed. Browserify bundles app files into bundle.js. An external vendor.js bundle speeds up the process.
'karma' is executed. In karma.conf.js, karma-browserify bundles app files again. It does not use an external bundle. Files are instrumented by Istanbul. Tests are executed by Karma. A coverage report is written by Istanbul.
When a file is changed, the bundle from 'build-dev' rebundles, saves the file as bundle.js, and reloads the connect server.
At the same time, karma-browserify rebundles the files again. It retests and rewrites the coverage report.

To me, this is wrong. As a result, I've been reading about somehow tying Karma into the rebundle process from 'build-dev', but I don't think it's going to work. Therefore, I'm going to look at using tape, angular-node, and coverify. I'll admit that I'm not really sure how it'll work.
Ideally, I want a 'build-test' task which performs all of the same steps from 'build-dev', but it also needs to run tests and generate a coverage report without the extra overhead of a second rebundling. I want it to be fast.
Links:

tape
angular-node
coverify
Substack suggests tape

Substack suggests "tape" because Substack made "tape" ;).
Honestly, I completely agree with you. As it stands, this approach is clunky and should be revised. Have you tried https://www.npmjs.com/package/gulp-karma at all? I'll give it a shot first chance I get.
That being said, I'm open to exploring tape, angular-node, and coverify. Angular-node and coverify are still fairly new, but I'm open to exploring them. That being said, I'm not completely sold on tape. It's mainly a personal preference, but I prefer the mocha / jasmine approach to testing more, largely because they seem to have more in common with most other test frameworks from other languages (like PHPUnit or TestNG -> Java).

Using a touchscreen...
I know that substack made tape. He also made browserify, so his suggestion seems valid to me
Gulp-karma doesn't let me pass a bundle into it or really control the steps of the process. It lets me pipe to karma, and the configuration is just a path to the karma.conf.js file.
I'm thinking that the coverage report should be generated not on every bundling but perhaps only every time a test file changes. Tests should be rerun every time an app js file changes, but maybe we could somehow only rerun the ones that reference files that changed.
I'll dig more into alternatives. I suggested tape/coverify because it seems like I can integrate them with gulp, whereas karma seems totally separated. Maybe I can still use jasmine/mocha-sinon-chai if I just find something to replace karma.

I was just giving you a hard time about tape. I honestly don't disagree with anything you have said. I'm also completely open to the prospect of using tape. My concern with it is that it's not as human readable as mocha/chai or jasmine, making it harder for a collaborator to come in and quickly grasp what's happening in the tests. Tests may require a fair bit of comments throughout. It does, however, seem like it would be considerably quicker than the current setup.
I'm also tempted, these days, to veer away from gulp or any other build tool in favour of straight scripts. There tends to be a lot of bloat with with gulp modules which slows things down.

I understand what you mean about gulp bloating things, and I've read similar sentiments before. Every tool I can think of has a command-line interface, so writing a gulpfile to glue everything together can be seen as a waste when you could just use some command-line arguments and pipe each output to the next input. I just spent some time reading about two approaches: npm run and make. They seem about the same. I'll spend some more time on it tomorrow.

My only hesitation at this point, with dropping gulp, is that this project has accumulated a decent following and I don't necessarily want to lose that. I'd definitely be interested in seeing how much of a speed difference something like "run" or "make" would be, but I'd like to continue with this current project, if only to keep the momentum going. For now I think it's worth looking into alternative testing options, though.

So after reading up more on tap / tape, I'm much more sold now. I'm going to investigate going that route instead of using karma.

Well. I haven't been doing anything more than reading lately, but I think Mocha could work without Karma as well.

Yeah, I found this as well: https://blog.engineyard.com/2015/client-side-javascript-project-gulp-and-browserify
There is an overwhelming number of options (as is the case with everything javascript). It's hard to choose which path to take. I'm also looking at https://theintern.github.io

Life has been busy lately...
I started reading more into angular-node, and it's pushing me away now. angular-node needs jsdom, and jsdom doesn't like Windows. It reminds me of when I first tried NodeJS a few years back, when every simple process involved a complicated list of "twelve easy steps."

Heh, my second kid was born last week, so I know all about life being busy ;).
Although jsdom is definitely cool in terms of what you can do with it, but the use case seems more functional than unit. The cool thing is that you can do functional unit tests, which is definitely neat.
Also, why are you developing in windows? Why not use a unix environment (Ubuntu VM, for example)? That's the more common dev environment these days unless you're working with microsoft languages like .NET.

I'm sure you know more about being busy than I do. ^.^"

jsdom was just something I wanted to use so that I could get angular-node. I wanted to write normal tests which require() Angular instead of running a Karma server or something. After some more thinking, it seems to me that smarter people than me probably decided to use things like Karma because that's the easiest way to get it working. Maybe instead of trying to get tests executing really fast, I should just accept that it'll take some time.

I use Windows because I like to play games... sometimes... when I can... Dual-booting is annoying, and surely a VM would be slow? I used Arch Linux on the laptop I had in college. Maybe I should set that up again. It might be less annoying to dual-boot now since I have an SSD.


Fix validation

I just cherry-picked the first 3 commits. the 4th doesn't compile (missing log)

@jdef looks like this PR can be closed then?

Yes.


[cuda]  improve maintainability for the version check

Just a minor change to improve the maintainability of the function.
There is already a CUDA_REQUIRED_VERSION, so use it to check it against the found version instead of using hardcoded values. (so whenever the required version changes it has to be set only in one place)
Also, use VERSION_LESS to compare versions.

Sorry I forgot about that, now it's done

Thanks for your contribution!


QC values thresholds should be adjustable

Is your feature request related to a problem? Please describe.
It would be good, if the user can adjust the thresholds for the QC metrics (eg Coverage, % Duplicates, Ts/Tv), for coloring them in red (or perhaps other colors), if they are below or above a defined threshold.
We currently have the problem, that for all our samples, the Ts/Tv ratio is colored in red, because it does not achieve the threshold value. However, I could not find this threshold and it should be adjustable to the local samples, because some QC values might be different from site to site.
Describe the solution you'd like
A possibly to add thresholds via an "options" page in the GUI or via a config file would be great.
Describe alternatives you've considered
n/a
Additional context

This should be adjustable on Project level


java autobuild overrides maven wrapper only-script distributionType

When using the maven wrapper distributionType only-script autobuild will override the wrapper:
[2024-07-13 09:29:53] [autobuild] > mvn -N org.apache.maven.plugins:maven-wrapper-plugin:3.3.0:wrapper -Dtype=bin

See https://github.com/ArloL/chorito/actions/runs/9918944552/job/27404224920 for a full log.
My guess is that autobuild checks for the presence of the maven-wrapper.jar because if I add an empty file back autobuild will not trigger the override. Maybe the check can be changed to check for maven-wrapper.properties or similar?
In the meantime I will revert back to the bin distributionType.

Looking at the change history of the autobuilder, it looks like this change was on purpose;  commit message:

The default kind of wrapper had changed to script-only in plugin 3.3.0. Also pin the plugin version to avoid unexpected surprises in future.

I assume there was an increase in reported failures after the wrapper switching its default behaviour from bin to script-only which was addressed by ensuring bin was used in all cases. The "autobuilder" is just a best effort approach, consisting of many heuristics to try to successfully build as many projects as possible. Note that the default switched to source-only quite recently (April 2024), so  using bin is probably still the best choice for most projects. In the future the balance will likely tip in favour of script-only once more and more project start relying on it.

Yes, the autobuilder assumes that if no maven-wrapper.jar is present then this may be because we are expected to acquire the jar ourselves somehow (often because we're working with source-control rules that don't allow storing a jar in-repo). However it doesn't recognise the augmented mvnw script that can do the download itself. It should at least be harmless, if a little annoying to revert what it does when working with a persistent checkout of the repository.


Immich added a new External Libraries feature

The app immich has added a new External Libraries feature with v1.79.0  which replaces the now deprecated read only gallery.
It may be worth re-evaluating the Existing Folders score.

I can confirm that the files are not duplicated in Immich database. Thumbnais and encoded videos will be generated.
In my case, 830GB of pictures/videos located in a NAS only need 210GB of space in the VM where immich is running without adjusting any thumbnail quality or encoding settings.


Obsoletion request: GO:0110154   RNA decapping and children

Please provide as much information as you can:

GO term ID and Label
GO:0110154  RNA decapping
Cleavage of the 5'-cap of an RNA.
and children
GO:0000290 deadenylation-dependent decapping of nuclear-transcribed mRNA
GO:0110155   NAD-cap decapping
GO:0110156   methylguanosine-cap decapping
GO:0031087 deadenylation-independent decapping of nuclear-transcribed mRNA
GO:0000290 deadenylation-dependent decapping of nuclear-transcribed mRNA
GO:0106289 negative regulation of deadenylation-dependent decapping of nuclear-transcribed mRNA
GO:0106288 regulation of deadenylation-dependent decapping of nuclear-transcribed mRNA
GO:1901835    positive regulation of deadenylation-independent decapping of nuclear-transcribed mRNA |
GO:1901834    regulation of deadenylation-independent decapping of nuclear-transcribed mRNA | regulates
GO:0036450   polyuridylation-dependent decapping of nuclear-transcribed mRNA
GO:1901835  positive regulation of deadenylation-independent decapping of nuclear-transcribed mRNA
GO:1901834  regulation of deadenylation-independent decapping of nuclear-transcribed mRNA	regulates

NOTE: We will need a MF term(s) for mRNA decapping activity first

Reason for deprecation Put an x in the appropriate box:

[x] The reason for obsoletion is that the data from the paper for which the term was requested can be accurately described using [appropriate GO term].

"Replace by" term (ID and label)
If all annotations can safely be moved to that term

"Consider" term(s) (ID and label)
Suggestions for reannotation

Are there annotations to this term?

How many EXP:

Are there mappings and cross references to this term? (InterPro, Keywords; check QuickGO cross-references section)

Is this term in a subset? (check the AmiGO page for that term)

Any other information

Checklist for ontology editor
Check term usage and metadata in Protégé

[ ] check term usage in the ontology
[ ] check internal mappings: RHEA, EC, MetaCyc
[ ] check subset usage
[ ] check taxon constraints

Check annotations

[ ] create Annotation Review spreadsheet

Annotation Review template: https://docs.google.com/spreadsheets/d/1izV3q1jnq_Ybg5MJ53O_b5BCTkGtjl_Fi-cOg3-7eBE/edit#gid=0

Instructions: https://wiki.geneontology.org/Annotation_Review_Request

[ ] create annotation review ticket: https://github.com/geneontology/go-annotation/issues/new?assignees=&labels=annotation+review&template=annotation-review-for-obsoletion.md&title=Review+annotations+to+[INSERT+GO+TERM]
[ ] find external mappings (via IEAS), include in obsoletion notice
[ ] tag curators in Annotation Review ticket. Responsible annotator for each group: https://github.com/geneontology/go-site/blob/master/metadata/group-contacts.csv

Notification

[ ] create obsoletion announcement
[ ] announce to GO friends<EMAIL_ADDRESS>[ ] paste the text in the ontology ticket

NOTE: We will need a MF term(s) for mRNA decapping activity first
See #24892
I believe we already have these:
. mRNA 5'-phosphatase activity (GO:0140818, EC:<IP_ADDRESS>)
. 5'-(N(7)-methylguanosine 5'-triphospho)-[mRNA] hydrolase activity (GO:0140933, EC:<IP_ADDRESS>)
. mRNA 5'-diphosphatase activity (GO:0034353, no EC)
. 5'-(N(7)-methyl 5'-triphosphoguanosine)-[mRNA] diphosphatase activity (GO:0140932, EC:<IP_ADDRESS>)
Maybe some term renames or added synonyms would make them clearer?
PMID:30345629 is a good review.

Thanks @sjm41 of course! ( I hadn't actually looked yet!) , but I will add decapping synonyms to these terms to make them  findable!

I'll close this one for now. I want to review some of the children like
deadenylation-independent decapping of nuclear-transcribed mRNA
deadenylation-dependent decapping of nuclear-transcribed mRNA
as it isn't clear if these are just describing different causal upstream processes, but this will be dealt with as I map out the pathways.


Excel process crashes after getting a response

We have a strange problem on only some computers:
After a response has been received, Excel completely crashes. It vanishes, restarts and offers to recover the previous file. On these machines, it happens each and every time.
On other machines, it works perfectly.
We added some logging that can be seen in the Immediate window, so we know that the response was already received successully.
A second or so after that, Excel just quits.
Of course, I have already googled a lot and found mostly other people having the same problem, but no fix.
Maybe you have an idea what I could try?
This is "Excel for Office 365 MSO (16.0.12130.20272) 64-bit" on "Windows 10 Pro 17134.1006".
In the event log I see this

I've noticed crashes when I try to parse large (unexpected) HTML pages rather than the expected JSON response. Usually I get those large HTML pages when the server send back the developer exception page at me.
You might want to dump the responses to a text file prior to parsing to see if you're getting what you think you're getting.

Pheeeew. That was an ordeal!
Finally figured out what triggered this.
It only happens with Excel 16.0.12130.20272
I had to use a VM to experiment and the Office Deployment Tool to have exactly that version installed.
Then a lot more of trial and error to find out that the following code triggered it.
I wrote a little helper function that should extract a field of a JSON response and return an empty string if it isn't included in the server response:
...
    Dim Response As WebResponse
    Dim Message As String

    Set Response = Client.Execute(Request)
    Set ResponseData = Response.Data
    Message = GetResponseMessage(ResponseData)
...
Function GetResponseMessage(ResponseData As Variant) As String
    On Error GoTo getResponseMessage_ErrorHandling
    GetResponseMessage = ResponseData("message")
    Exit Function

getResponseMessage_ErrorHandling:
    GetResponseMessage = ""
End Function

This works on all of our machines and many different Excel version, but always crashes on all the machines of our client's client' machines with that specific Excel version - regardless of whether the server response actually has the "message" field included.
I changed the code to the following and now it works:
...
    Dim Response As WebResponse
    Dim Message As String

    Set Response = Client.Execute(Request)
    Message = GetResponseMessage(Response)
...
Function GetResponseMessage(Response As WebResponse) As String
    On Error GoTo getResponseMessage_ErrorHandling
    GetResponseMessage = Response.Data("message")
    Exit Function

getResponseMessage_ErrorHandling:
    GetResponseMessage = ""
End Function

I noticed Excel crashed after using the request object for the second request. This fixed the issue::
Set Request = New WebRequest


chore: release main

:robot: I have created a release beep boop
codegen: 0.1.0
0.1.0 (2023-07-07)
Features

custom-layout-io: add way to derive custom layout IOs (#117) (61a8625)
docs: add docs for layout IO (#131) (551d65e)
gds-export: add GDS crate and utilities for accessing GDS layers (#87) (5cf11cd)
gds-export: implement GDS export of Substrate cells (#97) (ae5ca3d)
layer-api: add layer IDs to shapes (#85) (df7064d)
layer-api: initial layer API and codegen (#84) (42bd94c)
layer-families: implement layer families and clean up codegen (#127) (06f50b8)
layout-io: initial layout port API implementation (#111) (ecc8838)
layout-ports: initial implementation of layout port traits (3c0527a)
mos: add standard 4-terminal MosIo (8678716)
organization: rename substrate to substrate_api, set up codegen crate (#67) (e07f099)
pdk: add PDK trait and update context (#68) (a8fbd14)
pdks: implement supported_pdks macro and add examples (#72) (5f4312f)
proc-macros: add derive Block proc macro (#151) (e2c2f02)
proc-macros: allow missing docs on generated structs (8678716)
proc-macros: macros respect field and struct visibilities (8678716)
proc-macros: proc macros find substrate crate location (#125) (8678716)
schematic: nested node and instance access (#134) (3d0e9ce)
schematics: implement node naming trees, with codegen (#105) (5ef8e4b)
schematics: implement proc macro to derive AnalogIo (#99) (2320c99)

Bug Fixes

deps: update rust crate syn to v2 (#79) (eee3593)

Dependencies

The following workspace dependencies were updated

dependencies

substrate_api bumped from 0.0.0 to 0.1.0

dev-dependencies

substrate bumped from 0.0.0 to 0.1.0

config: 0.1.0
0.1.0 (2023-07-07)
Features

config: config merging and parsing functionality (#40) (13c8925)
docs: add missing documentation to config crate (#55) (cf10436)
organization: rename substrate to substrate_api, set up codegen crate (#67) (e07f099)
simulation: support transient simulation in spectre (#133) (4605862)

Bug Fixes

config: add config to release manifest and fix version (#41) (b7097f5)
deps: fix dependencies and documentation (#66) (a60ffc6)
docs: fix additional clippy errors and missing docs (#56) (f76a169)
docs: fix broken links and check docs in CI (#59) (13dc7a5)

diagnostics: 0.1.0
0.1.0 (2023-07-07)
Features

api: initial SCIR API definition (#51) (c175a48)

Bug Fixes

docs: fix broken links and check docs in CI (#59) (13dc7a5)
release: publish diagnostics and add tests to manifest for versioning (cc13215)

gds: 0.1.0
0.1.0 (2023-07-07)
Features

gds-export: add GDS crate and utilities for accessing GDS layers (#87) (5cf11cd)
gds-export: implement GDS export of Substrate cells (#97) (ae5ca3d)

Bug Fixes

deps: update rust crate chrono to 0.4.26 (#88) (013d937)
deps: update rust crate num-derive to 0.4 (#121) (e7f491e)
deps: update rust crate serde_yaml to 0.9 (#93) (3947a23)
deps: update rust crate textwrap to 0.16.0 (#94) (84231b7)
deps: update rust crate toml to 0.7.5 (#91) (779c51f)

geometry: 0.2.0
0.2.0 (2023-07-07)
Features

blackboxing: support Substrate and SCIR blackboxes (#135) (049a598)
tests: add SCIR and SPICE netlister blackbox tests (049a598)

opacity: 0.1.0
0.1.0 (2023-07-07)
Features

blackboxing: support Substrate and SCIR blackboxes (#135) (049a598)
tests: add SCIR and SPICE netlister blackbox tests (049a598)

pathtree: 0.1.0
0.1.0 (2023-07-07)
Features

blackboxing: support Substrate and SCIR blackboxes (#135) (049a598)
simulation: access nested nodes without strings in simulation (#139) (ed7989c)
tests: add SCIR and SPICE netlister blackbox tests (049a598)

scir: 0.1.0
0.1.0 (2023-07-07)
Features

api: initial SCIR API definition (#51) (c175a48)
blackboxing: support Substrate and SCIR blackboxes (#135) (049a598)
buses: add support for 1D SCIR buses (#57) (162889c)
netlisting: initial implementation of SPICE netlister (#102) (9125446)
node-naming: create internal, named signals of any schematic type (#118) (1954bb9)
pdks: example instantiation of PDK-specific MOS (#112) (bbac00c)
schematics: export Substrate schematics to SCIR (#110) (28115f0)
simulation: access nested nodes without strings in simulation (#139) (ed7989c)
simulation: simplify SCIR paths for data access (#143) (d42e6f9)
simulation: support transient simulation in spectre (#133) (4605862)
tests: add SCIR and SPICE netlister blackbox tests (049a598)

Bug Fixes

netlisting: fix whitespace issues (9125446)

Dependencies

The following workspace dependencies were updated

dependencies

diagnostics bumped from 0.0.0 to 0.1.0
opacity bumped from 0.0.0 to 0.1.0

sky130pdk: 0.1.0
0.1.0 (2023-07-07)
Features

corners: add API for process corners (#141) (a61b15a)
layer-families: implement layer families and clean up codegen (#127) (06f50b8)
layers: initial layer set for SKY130 PDK (#120) (1ea5a7e)
mos: add sky130pdk transistor blocks (#126) (3e9ee79)
mos: add standard 4-terminal MosIo (8678716)
proc-macros: allow missing docs on generated structs (8678716)
proc-macros: macros respect field and struct visibilities (8678716)
proc-macros: proc macros find substrate crate location (#125) (8678716)
sky130pdk: add Sky130Pdk struct definition (#124) (06ced7a)

Dependencies

The following workspace dependencies were updated

dependencies

substrate bumped from 0.0.0 to 0.1.0
spectre bumped from 0.0.0 to 0.1.0

spectre: 0.1.0
0.1.0 (2023-07-07)
Features

blackboxing: support Substrate and SCIR blackboxes (#135) (049a598)
corners: add API for process corners (#141) (a61b15a)
inverter-design: initial inverter design logic (#144) (911a376)
netlisting: implement basic Spectre netlister (#130) (506837f)
scir: uniquify names when exporting to SCIR (#148) (29c2f72)
simulation: access nested nodes without strings in simulation (#139) (ed7989c)
simulation: simplify SCIR paths for data access (#143) (d42e6f9)
simulation: support transient simulation in spectre (#133) (4605862)
simulation: testbench schematic components (#136) (97e6b0f)
tests: add SCIR and SPICE netlister blackbox tests (049a598)

Bug Fixes

netlisting: update release manifests (506837f)

Dependencies

The following workspace dependencies were updated

dependencies

scir bumped from 0.0.0 to 0.1.0
substrate bumped from 0.0.0 to 0.1.0
opacity bumped from 0.0.0 to 0.1.0

spice: 0.0.0
Dependencies

The following workspace dependencies were updated

dependencies

scir bumped from 0.0.0 to 0.1.0
opacity bumped from 0.0.0 to 0.1.0

substrate: 0.1.0
0.1.0 (2023-07-07)
Features

blackboxing: support Substrate and SCIR blackboxes (#135) (049a598)
docs: add code examples to documentation (#65) (bfafd05)
layout-api: initial implementation of layout API (#61) (c4cdac7)
mos: add sky130pdk transistor blocks (#126) (3e9ee79)
mos: add standard 4-terminal MosIo (8678716)
organization: rename substrate to substrate_api, set up codegen crate (#67) (e07f099)
pdk: add PDK trait and update context (#68) (a8fbd14)
pdks: implement supported_pdks macro and add examples (#72) (5f4312f)
proc-macros: allow missing docs on generated structs (8678716)
proc-macros: macros respect field and struct visibilities (8678716)
proc-macros: proc macros find substrate crate location (#125) (8678716)
simulation: simplify SCIR paths for data access (#143) (d42e6f9)
simulation: support transient simulation in spectre (#133) (4605862)
simulation: testbench schematic components (#136) (97e6b0f)
tests: add SCIR and SPICE netlister blackbox tests (049a598)

Bug Fixes

deps: fix dependencies and documentation (#66) (a60ffc6)
re-exports: move all re-exports to substrate (#132) (8b3d867)

Dependencies

The following workspace dependencies were updated

dependencies

geometry bumped from 0.1.0 to 0.2.0
codegen bumped from 0.0.0 to 0.1.0
substrate_api bumped from 0.0.0 to 0.1.0
scir bumped from 0.0.0 to 0.1.0

substrate_api: 0.1.0
0.1.0 (2023-07-07)
Features

blackboxing: support Substrate and SCIR blackboxes (#135) (049a598)
cache: add initial implementation of in-memory caching (#150) (2b26077)
corners: add API for process corners (#141) (a61b15a)
custom-layout-io: add way to derive custom layout IOs (#117) (61a8625)
docs: add docs for layout IO (#131) (551d65e)
docs: add documentation for schematic API (#128) (b78b2e6)
gds-export: add GDS crate and utilities for accessing GDS layers (#87) (5cf11cd)
gds-export: implement GDS export of Substrate cells (#97) (ae5ca3d)
generation: generating cells no longer blocks unless absolutely necessary (#140) (1d33dd0)
layer-api: add layer IDs to shapes (#85) (df7064d)
layer-api: initial layer API and codegen (#84) (42bd94c)
layer-families: implement layer families and clean up codegen (#127) (06f50b8)
layout-io: implement exporting layout IO to GDS (#129) (e9973a0)
layout-io: initial layout port API implementation (#111) (ecc8838)
layout-ports: initial implementation of layout port traits (3c0527a)
mos: add sky130pdk transistor blocks (#126) (3e9ee79)
node-naming: cell builder maintains node name map (#109) (d9cf26e)
node-naming: create internal, named signals of any schematic type (#118) (1954bb9)
node-naming: priority system for determining node names (#115) (fe746e2)
organization: rename substrate to substrate_api, set up codegen crate (#67) (e07f099)
pdks: example instantiation of PDK-specific MOS (#112) (bbac00c)
pdks: implement supported_pdks macro and add examples (#72) (5f4312f)
schematic: nested node and instance access (#134) (3d0e9ce)
schematics: add instantiate connected API function (#122) (b36eab6)
schematics: allow indexing/slicing of arrays (#100) (363344e)
schematics: export Substrate schematics to SCIR (#110) (28115f0)
schematics: implement Array<T> HW type (#98) (6625538)
schematics: implement flip for Direction (#96) (d03fa82)
schematics: implement node naming trees, with codegen (#105) (5ef8e4b)
schematics: implement proc macro to derive AnalogIo (#99) (2320c99)
schematics: initial schematic API implementation (#86) (332d5d7)
schematics: support port directions (#95) (1c660f7)
scir: uniquify names when exporting to SCIR (#148) (29c2f72)
simulation: access nested nodes without strings in simulation (#139) (ed7989c)
simulation: initial simulator API (#80) (249c557)
simulation: simplify SCIR paths for data access (#143) (d42e6f9)
simulation: support transient simulation in spectre (#133) (4605862)
simulation: testbench schematic components (#136) (97e6b0f)
tests: add SCIR and SPICE netlister blackbox tests (049a598)
transformed-view: add TransformedView trait for lazy evaluation of collections (70ba0bb)
uniquify: create uniquify crate for assigning unique names (#147) (d4b83be)
waveforms: clocked digital waveform builder (5f5beab)
waveforms: edge and transition APIs (5f5beab)
waveforms: implement Waveform and WaveformRef (5f5beab)
waveforms: time-domain waveform API (#145) (5f5beab)

Bug Fixes

node-naming: implement flatten for name trees (#108) (45baa4d)
re-exports: move all re-exports to substrate (#132) (8b3d867)

Dependencies

The following workspace dependencies were updated

dependencies

geometry bumped from 0.1.0 to 0.2.0
gds bumped from 0.0.0 to 0.1.0
opacity bumped from 0.0.0 to 0.1.0
scir bumped from 0.0.0 to 0.1.0
pathtree bumped from 0.0.0 to 0.1.0
uniquify bumped from 0.0.0 to 0.1.0

dev-dependencies

substrate bumped from 0.0.0 to 0.1.0

tests: 0.0.0
Dependencies

The following workspace dependencies were updated

dependencies

geometry bumped from 0.1.0 to 0.2.0
substrate bumped from 0.0.0 to 0.1.0
scir bumped from 0.0.0 to 0.1.0
spectre bumped from 0.0.0 to 0.1.0
sky130pdk bumped from 0.0.0 to 0.1.0

uniquify: 0.1.0
0.1.0 (2023-07-07)
Features

scir: uniquify names when exporting to SCIR (#148) (29c2f72)
uniquify: create uniquify crate for assigning unique names (#147) (d4b83be)

This PR was generated with Release Please. See documentation.

:robot: Release is at https://github.com/substrate-labs/substrate2/releases/tag/codegen-v0.1.0 :sunflower:

:robot: Release is at https://github.com/substrate-labs/substrate2/releases/tag/config-v0.1.0 :sunflower:

:robot: Release is at https://github.com/substrate-labs/substrate2/releases/tag/diagnostics-v0.1.0 :sunflower:

:robot: Release is at https://github.com/substrate-labs/substrate2/releases/tag/gds-v0.1.0 :sunflower:

:robot: Release is at https://github.com/substrate-labs/substrate2/releases/tag/geometry-v0.2.0 :sunflower:

:robot: Release is at https://github.com/substrate-labs/substrate2/releases/tag/opacity-v0.1.0 :sunflower:

:robot: Release is at https://github.com/substrate-labs/substrate2/releases/tag/pathtree-v0.1.0 :sunflower:

:robot: Release is at https://github.com/substrate-labs/substrate2/releases/tag/uniquify-v0.1.0 :sunflower:

:robot: Release is at https://github.com/substrate-labs/substrate2/releases/tag/spice-v0.0.0 :sunflower:

:robot: Release is at https://github.com/substrate-labs/substrate2/releases/tag/sky130pdk-v0.1.0 :sunflower:

:robot: Release is at https://github.com/substrate-labs/substrate2/releases/tag/scir-v0.1.0 :sunflower:

:robot: Release is at https://github.com/substrate-labs/substrate2/releases/tag/substrate-v0.1.0 :sunflower:

:robot: Release is at https://github.com/substrate-labs/substrate2/releases/tag/substrate_api-v0.1.0 :sunflower:

:robot: Release is at https://github.com/substrate-labs/substrate2/releases/tag/tests-v0.0.0 :sunflower:

:robot: Release is at https://github.com/substrate-labs/substrate2/releases/tag/spectre-v0.1.0 :sunflower:


intersectionEnv: is the has_arrived function always checking for if the agents has arrived in inner left lane?

https://github.com/eleurent/highway-env/blob/742ce2e69bf00169f2b250e0388c6b9926c026ec/highway_env/envs/intersection_env.py#L260
Can this be the reason why the agent  takes a u-turn after it reached the end of the lane and joined the outer lane?
For recreating
I was using it with multiple agents (4)
0 initial vehicles count
changed the terminal condition to all
https://github.com/eleurent/highway-env/blob/742ce2e69bf00169f2b250e0388c6b9926c026ec/highway_env/envs/intersection_env.py#L99
increased duration (20)

The agent makes the u-turn at the end of the lane because there is no next lane to follow. By default, in this situation, the agent is configured to follow the closest other lane, which is the one leading back to the intersection.
Normally, that does not happen because (in the single-player setting) the has_arrived condition is raised, which terminates the episode
I just realised there might be a bug in has_arrived(): it seems like I only test the west direction, and not other ones (probably because originally I was mostly testing the left turn for a single-agent coming from the south. I'll fix it.
Even with a fixed has_arrived condition, this will probably still happen if you set the terminal condition to all, since the first agents to arrive will have to keep driving. I'm not sure what a good solution to that must be... make the exit lanes longer? stop the agents?

Even with a fixed has_arrived condition, this will probably still happen if you set the terminal condition to all, since the first agents to arrive will have to keep driving. I'm not sure what a good solution to that must be... make the exit lanes longer? stop the agents?

would it make sense to remove the agent out of the scene (from the list of controlled vehicles) on arriving at the destination?


Make save_func not a function of the integrator

This PR makes the save_func not a function of the integrator, which alleviates a circular dependency upon initializing the integrator. A peel off from #115.

bors r+


add bcruddy to the authors file and contributors section

Just adding myself to contributors/authors and putting them in alphabetical order

This seems like one that doesn't need a +2 😉

Yes :) However we are 2. Aren't we? 😄


Move authentication down to Terraform

Description of your changes
Fixes #613
I have:

[x] Run make reviewable test to ensure this PR is ready for review.

How has this code been tested
Tested with the following provider configurations on an IRSA-enabled EKS cluster:

IAM User access key
IAM User access key (uptest run: https://github.com/upbound/provider-aws/actions/runs/4524510609)
IRSA
IRSA with an AssumeRole chain of length 1
IRSA with an AssumeRole chain of length 2
WebIdentity
WebIdentity with RoleSessionName configuration
WebIdentity with an AssumeRole chain of length 1
WebIdentity with an AssumeRole chain of length 2

/test-examples="examples/ec2/vpc.yaml"

/test-examples="examples/ec2/vpc.yaml"

/test-examples="examples/ec2/vpc.yaml"

Tested with the following provider configurations:

IAM User access key
IRSA
IRSA with an AssumeRole chain of length 1
IRSA with an AssumeRole chain of length 2
WebIdentity -> Issue detected (empty string is not considered as a valid session name by Terraform)

/test-examples="examples/ec2/vpc.yaml"

Because GitHub has now degraded availability for PRs and issues, leaving a review comment as a regular one:
internal/clients/aws.go:108
Let's try to preserve the old behavior and not introduce a breaking change here:
https://github.com/upbound/provider-aws/blob/c36cfcd775aadabfa65533014d94cffcceb0e88e/internal/clients/provider_config.go#L308

/test-examples="examples/ec2/vpc.yaml"


Initial implementation of the serializer for Table API Support.

Table API support requires conversion of Row Data to Protobuf for writing to BigQuery via the Storage Write API. This PR deals with the initial implementation for the same.
Note:

BIGNUMERIC , DATETIME and TIME Data is not supported yet. (Will be done in future, since Read needs to be fixed.)
The serializer will not support INTERVAL_YEAR_MONTH, INTERVAL_DAY_TIME, MAP, MULTISET, NULL, SYMBOL, RAW, DISTINCT_TYPE, STRUCTURED_TYPE, TIMESTAMP_WITH_TIME_ZONE in the MVP release of Table API Support.

/gcbrun

@clmccart requesting your review on this PR. Thanks!

@yerramalli requesting your review on this PR. Thanks!


Executing a parsed query documents raises an error

Invoking GraphQL::Schema.execute with an already parsed GraphQL document results in an error like this:
     TypeError:
       nil can't be coerced into Float
     # /Users/jturkel/.rvm/gems/ruby-2.5.3@bugs/gems/graphql-metrics-4.0.2/lib/graphql/metrics/instrumentation.rb:39:in `-'
     # /Users/jturkel/.rvm/gems/ruby-2.5.3@bugs/gems/graphql-metrics-4.0.2/lib/graphql/metrics/instrumentation.rb:39:in `after_query'
     # /Users/jturkel/.rvm/gems/ruby-2.5.3@bugs/gems/graphql-1.10.14/lib/graphql/execution/instrumentation.rb:82:in `public_send'
     # /Users/jturkel/.rvm/gems/ruby-2.5.3@bugs/gems/graphql-1.10.14/lib/graphql/execution/instrumentation.rb:82:in `block in call_after_hooks'
     # ...

It looks like the problem is GraphQL::Metrics::Tracer relies on the lex trace event to capture the query start time resulting in an unexpected nil in GraphQL::Metrics::Instrumentation#after_query. Perhaps in this scenario GraphQL::Metrics::Tracer should use the validate trace hook to set the query start time? I'd be happy to put up a PR after nailing down the general approach.
Here's a reproducible test case:
require 'bundler/inline'

gemfile(true) do
  source 'https://rubygems.org'
  gem 'graphql', '1.10.14'
  gem 'graphql-metrics', '4.0.2'
  gem 'activesupport'
  gem 'rspec'
end

require 'active_support/all'
require 'rspec/autorun'

class QueryType < GraphQL::Schema::Object
  field :ping, String, null: false

  def ping
    'ok'
  end
end

class SimpleAnalyzer < GraphQL::Metrics::Analyzer
  def query_extracted(metrics)
    # no-op
  end

  def field_extracted(metrics)
    # no-op
  end

  def argument_extracted(metrics)
    # no-op
  end
end

class TestSchema < GraphQL::Schema
  use GraphQL::Execution::Interpreter
  use GraphQL::Analysis::AST

  query_analyzer SimpleAnalyzer
  instrument :query, GraphQL::Metrics::Instrumentation.new
  tracer GraphQL::Metrics::Tracer.new

  query QueryType
end

describe 'GraphQL::Schema#execute' do
  it 'accepts query documents' do
    query = <<~GRAPHQL
      query {
        ping
      }
    GRAPHQL

    query_document = GraphQL.parse(query)

    expected = {
      data: {
        ping: 'ok'
      }
    }

    result = TestSchema.execute(document: query_document)
    expect(result.to_json).to eq(expected.to_json)
  end
end

Perhaps in this scenario GraphQL::Metrics::Tracer should use the validate trace hook to set the query start time?

That seems like a good compromise. Just check if the start time has already been set or not within validate, if not, set it.

+1 that sounds good!


Other pictures flickers when a media is selected from contributions list

Summary:
I always knew I was seeing flickering images when I choose an item from contrib list and trying to go to Media Details Fragment. While I was testing another bug, I recognized on click to an object from contrib list, rebuildCatList method from MediaDetailsFragment is called for previous images too. And I see previous image is appeared during my debug steps.
This issue should be investigated.

Pinging @ashishkumar468

Hey @neslihanturan Are you still facing this issue?

yes, this problem is still there and will stay there as viewpager is used and what is happening here is you if click on first item then it should be okay and no flickering will occur but when going to 2nd and beyond there is also a scrolling part taking place, what we can do here is adding a smooth scrolling so it looks nice plus user will able to get to know that they can scroll left and right.

can i work on this issue

please check Fixes #3879 #4334
solved this issue also added a little animation when opening detail screen and going back (because it looks like that was the developer trying to do while adding delay) and also minor optimization to code repetition and if else statement.
let me know if you want me to remove the animation

@vinayak0505 Would you mind removing animations from this screen? Thanks! In particular I observe that in the latest master branch, when opening the app the contributions appear from the right. While animation can be good sometimes, making the main content appear from the side when starting the app is not good: the content should appear immediately. I think we should remove animations from this screen as it is quite bug-prone.

okay so the real problem is viewpager get the complete data , not just for the single screen , so when you click on an item it takes time to load therefore the app crashes with error code like expected pager adapter count 60 found 70 pager id(its weird why there always a gap of 10 and also i am not able to reproduce this error as its not crashing on my device), so someone came up with the solution that we sould load the 1st item in adapter, wait for 5 millisec and then go to the item selected with causes flickering , the app was not crashing most of the time as it would wait for 5 millisec and if the dataloads bingo if not then the app is like its your problem , creating a method that can notifiy when the data loading is complete or removing viewpager with normal detail screen or atleast converting it to viewpager 2 , what i am soing now is creating a background thread looping till the data loads to the selected item and when it does it will it go to that page(rather than just waiting for 5 millisec), ill make a pull request , i have tried loading 80 images and i still dont face any issue so youll have to test this if this works it will also close the bug #4283
please check #4336 i am not able to reproduce this error of crashing so i have only solve them using the error you are telling and logcat you send, so you will have to test it on your device

Thanks @nicolas-raoul


Bot should notify maintainers when a backer creates an issue

I believe it would be beneficial if we added a notify option to the configuration which would modify the behaviour of the app in such a manner that would notify maintainers once someone who is backing the project submits an issue or a PR.

Perhaps an email could be sent for each item labelled as "priority".
But the address to which it is sent should not be in the repository settings (for privacy). I think that as a collective you should be able to configure the maintenance address in the frontend of opencollective.

Open Collective bot is discontinued.


[7.x] Add method to support clearing registered paths

This change should be fairly self-explanitory.
As for the motivation, I'm working on a multi-database SAAS app that needs to run migrations programmatically when a "tenant" (and its associated database) is created.  I'm already able to dynamically add the "tenant migrations" path to the migrator, which works fine,  However, all other registered migrations (including those registered with loadMigrationsFrom() in other packages) will still run.
If there's a better approach, I'd love to hear it.
Additionally, I'm happy to write a test for this if you wish, but wasn't sure where to put it.  Guidance would be appreciated.

Thanks for your pull request to Laravel!
Unfortunately, I'm going to delay merging this code for now. To preserve our ability to adequately maintain the framework, we need to be very careful regarding the amount of code we include.
If possible, please consider releasing your code as a package so that the community can still take advantage of your contributions!
If you feel absolutely certain that this code corrects a bug in the framework, please "@" mention me in a follow-up comment with further explanation so that GitHub will send me a notification of your response.

@taylorotwell , I understand that's a copy/paste response, but I don't see how this could ever be released as a separate package.  One would have to completely replace the migrator instance just to add a single method.  Seems a bit overkill for a package.
If you don't want to add it, that's perfectly fine.  I can find a different way to do what I'm trying to do.  I just figured this would be easier.
Thank you for your time.


getTenantAPIs() is not constant time

Because we don't have a mapping tenant -> api getTenantAPIs has to get all the APIs in redis, deserialize the json and check if it matches the tenantId.. This will be super slow if we take this into production. here

We do look up the list of APis by tenant, so while still not constant time, should be significantly more performant than sorting through every API in the datastore.
Am I missing something?

Ah, I think you're referencing the management interface. The actual gateway path does not have this issue.
@songs what was the reasoning behind the current implementation?

Also, this mgmt interface isn't used in a typical production scenario. Only in the standalone gateway.

@mhamann As you mentioned, this management interface is used only in the standalone gateway, and so there aren't as many tenants/APIs to account for. It was implemented this way so that how redis is structured doesn't deviate too much from how it's handled in production.
However, it is true that this isn't the most optimal, and can be fixed by adding additional redis keys, as suggested by @taylorking.

FIxed via #113


Trouble connecting Bookings to Facebook

I was directed to this page for instructions to connect Bookings to my Facebook account. It does not have instructions and I cannot find them anywhere else.
Can you provide instructions to connect Bookings to Facebook?
Thanks,
Andy

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: adb53eb7-85a1-6aa0-dfd1-e2b54586008a
Version Independent ID: 5532470a-a47d-e3bd-4e74-02b2c822cae1
Content: Microsoft Bookings
Content Source: microsoft-365/bookings/bookings-overview.md
Service: bookings
GitHub Login: @kwekuako
Microsoft Alias: kwekua

@alathrop  thanks for the feedback. We updated the page with a banner stating that connecting to Facebook is no longer supported. The button should be removed from the Bookings app shortly. Thanks.


[WIP] Pdf templates

JIRA link (if applicable)
HTML template
PDF form
Change description
Attempt to kick off generalising pdf generator with any template
Does this PR introduce a breaking change? (check one with "x")
[ ] Yes
[x] No

I think we should wait until we know if PDF templates are going to be used


Suggestion: disable the automatic start of the plugin by setting timeout = 0

Hi,
First of all, the plugin really looks cool, great job!
One thing: from what I understand, timeout is the "screensaver timeout". I want to disable the automatic screen saver and only do it with the :Donut command.
I thought setting timeout = 0 would achieve this, however, when I do that, it will be running forever, and I can't stop it.
I think it would be nice to have this feature similar to "normal" screensavers that can be disabled

Hi!

First of all, the plugin really looks cool, great job!

Thanks!

One thing: from what I understand, timeout is the "screensaver timeout". I want to disable the automatic screen saver and only do it with the :Donut command.
I thought setting timeout = 0 would achieve this, however, when I do that, it will be running forever, and I can't stop it.
I think it would be nice to have this feature similar to "normal" screensavers that can be disabled

That's a reasonable suggestion. I added it in the latest release. If you experience any bugs related to this, feel free to open up a new issue.