Neo4j support

This feature requests serves as a central place to discuss development and progress for the Neo4j connector.

Would be great

+1

Also interested to contribute on this.

When is support expected to be released? a week? a month?
Thanks 👍

I'm building an server that requires some machine learning features entirely depends on neo4j.
I also have been using prisma from its alpha state.
Nothing will be better if prisma supports neo4j!

Yea so I've been getting a bit more into this now
The first thing to understand is that the functionality we're all looking for already exists in an abstract sense via GRANDstack
... let me explain.
So I started out using Prisma backed by postgres.  Defined my schema, built a social network application and got all hydrated with user data.  THEN  I discover Neo4j.. a friggen beautiful piece of art in the form of a Database
As it turns out in nearly entirely turn-key "just works" fashion you can both import your data from Postgres --> Neo4j (https://neo4j.com/developer/neo4j-etl/), and then use Neo4j-->GraphQL schema introspection from GRANDstack (specifically https://github.com/neo4j-graphql/neo4j-graphql-js ) and have nearly functionally equivalent GraphQL endpoint which is backed by your new Neo4j database in very little time (for me it took a couple days for a fairly involved application) this is obviously a punchline.
... as such I would recommend anybody here serious about creating the Neo4j connector with Prisma should try out GRANDStack and get some experience looking through that lens in order to see GRANDstack's "big picture" juxtaposed with Nexus's
Now I've come to understand that it's not a mutually exclusive GRANDStack - or - Nexus Framework decision.  Fundamentally each project has a distinct area of focus.
That said they certainly overlap.  For instance both handle schema definition/construction.  However Nexus achieves this via code.  And the neo4j-graphql-js graphql middleware is bootstrapped via GraphQL SDL.  Both result in a graphql endpoint that I can generate TypeScript types from for nearly compatible typesafe usage on my frontend.
Anyways this is just an intro.  I suppose it's also worth noting that I was running a Kubernetes cluster for my Nexus Framework endpoint and Prisma Postgres instance, and now I'm literally running a single google cloud run serverless service which hosts the neo4j-graphql-js middleware.  The Neo4j instance I outsourced to their own Neo4j managed "Aura" (provisioned through GCP)
I sense intuitively that this integration should/will happen.  Graph-native databases are fundamental staple of modern software bag-o-tricks, meaningful Nexus-Prisma-GRANDstack integration is a must.  As I get deeper into GRANDstack I'd like to really seriously ask the question what this Neo4j connector could look like.  Especially the "WHY?"  And then the "HOW?" and ultimately the concrete "WHAT?"
IF NOTHING ELSE CLICK TO SEE THE DATA VIS I GOT OUT OF THE BOX FOR VERY LITTLE TIME INVESTED

In grand stack the graphql is discovered from the database contents, can we also generate the prisma schema from that?

any update on this one?

In my opinion, the key differences between Prisma’s existing supported connectors and those Graph types such as; Neo4j, ArangoDB, and Amazons Neptune is the way you think and model your data plus support for more sophisticated features such as; Relationship properties, pattern matching, and recommendations.
Neo4j has already done a great job of showcasing how to generate a GraphQL API, the Cypher(Neo4j’s query language) behind that, and even exposing an ORM/OGM(Object Graph Mapper) for the same ecosystem, we should look at their efforts for inspiration here https://github.com/neo4j/graphql.
Looking further into modeling, given we take the schema from Prisma’s data model docs https://www.prisma.io/docs/concepts/components/prisma-schema/data-model:
datasource db {
  provider = "postgresql"
  url      = env("DATABASE_URL")
}

generator client {
  provider = "prisma-client-js"
}

model User {
  id      Int      @id @default(autoincrement())
  email   String   @unique
  name    String?
  role    Role     @default(USER)
  posts   Post[]
  profile Profile?
}

model Profile {
  id     Int    @id @default(autoincrement())
  bio    String
  user   User   @relation(fields: [userId], references: [id])
  userId Int
}

model Post {
  id         Int        @id @default(autoincrement())
  createdAt  DateTime   @default(now())
  title      String
  published  Boolean    @default(false)
  author     User       @relation(fields: [authorId], references: [id])
  authorId   Int
  categories Category[] @relation(references: [id])
}

model Category {
  id    Int    @id @default(autoincrement())
  name  String
  posts Post[] @relation(references: [id])
}

enum Role {
  USER
  ADMIN
}

This schema represents a relational model using a SQL-based provider, and for us to change this to represent Neo4j’s labeled property graph model there are a few key changes that would need to happen.
Firstly, Relationships are First Class
There are no foreign keys or joins in Neo4j and or Graphs in general. Instead, graphs embrace the concept of a relationship and in Neo4j’s case, they are ‘First Class Entities'. Let's take a subset of the data model above and dissect it to see how it would compare to that If we were using Neo4j.
model User {
  id      Int      @id @default(autoincrement())
  email   String   @unique
  name    String?
  role    Role     @default(USER)
  posts   Post[]
}

model Post {
  id         Int        @id @default(autoincrement())
  createdAt  DateTime   @default(now())
  title      String
  published  Boolean    @default(false)
  author     User       @relation(fields: [authorId], references: [id])
  authorId   Int
}

In this subset we define a @relation(fields: [authorId], references: [id]) on the Post model, this relationship is ‘linked’ together by appending the corresponding authorId field on a Post, and at runtime, when you want to query a user along with its posts, a join populates the nested data.
Previously mentioned that in Neo4j a join does not exist and instead, you should, when using graphs, rely on the relationships and the mechanisms for querying them. Here I will show you an abstract model representing the same schema above using a labeled property graph:

Notice, firstly, that a Post now does not contain any authorId field, and instead, we introduce a ‘label’ on the relationship itself HAS_POST.
Below I introduce you to my idea of what the Prisma schema would look like for supporting Neo4j.
Firstly, the keyword model is used to represent a Node, just like how it represents a Document using Mongo or a Tabel otherwise.
Then, I introduce the@edge attribute, it will work similarly to how we already define relationships in Prisma. I have decided to recommend using something new such as this attribute over using the existing @relation because semantically this will make it very clear that you are working with a graph plus it will provide a platform to support more sophisticated features such as relationship properties.
model User {
  id      Int      @id @default(autoincrement())
  email   String   @unique
  name    String?
  role    Role     @default(USER)
  posts   Post[]
}

model Post {
  id         Int        @id @default(autoincrement())
  createdAt  DateTime   @default(now())
  title      String
  published  Boolean    @default(false)
  author     User       @edge(type: "HAS_POST", direction: IN)
}

The key differences are that; authorId field no longer exists on a Post, and the @edge directive is used where it specifies a type and a direction.
So in all if we wanted to model this entire schema into an abstract graph model we could represent it as so:

Then this would translate into the following Prisma schema:
datasource db {
  provider = "neo4j"
  url      = env("DATABASE_URL")
}

generator client {
  provider = "prisma-client-js"
}

model User {
  id      Int      @id @default(autoincrement())
  email   String   @unique
  name    String?
  role    Role     @default(USER)
  posts   Post[]
  profile Profile?
}

model Profile {
  id     Int    @id @default(autoincrement())
  bio    String
  user   User   @edge(type: "HAS_PROFILE", direction: IN)
}

model Post {
  id         Int        @id @default(autoincrement())
  createdAt  DateTime   @default(now())
  title      String
  published  Boolean    @default(false)
  author     User       @edge(type: "HAS_POST", direction: IN)
  categories Category[] @edge(type: "IN_CATEGORY", direction: OUT)
}

model Category {
  id    Int    @id @default(autoincrement())
  name  String
  posts Post[] @edge(type: "IN_CATEGORY", direction: IN)
}

enum Role {
  USER
  ADMIN
}

Diff:
datasource db {
-  provider = "postgresql"
+  provider = "neo4j"
  url      = env("DATABASE_URL")
}

generator client {
  provider = "prisma-client-js"
}

model User {
  id      Int      @id @default(autoincrement())
  email   String   @unique
  name    String?
  role    Role     @default(USER)
  posts   Post[]
  profile Profile?
}

model Profile {
  id     Int    @id @default(autoincrement())
  bio    String
-  user   User   @relation(fields: [userId], references: [id])
+  user   User   @edge(type: "HAS_PROFILE", direction: IN)
-  userId Int
}

model Post {
  id         Int        @id @default(autoincrement())
  createdAt  DateTime   @default(now())
  title      String
  published  Boolean    @default(false)
-  author     User       @relation(fields: [authorId], references: [id])
+  author     User       @edge(type: "HAS_POST", direction: IN)
-  authorId   Int
-  categories Category[] @relation(references: [id])
+  categories Category[] @edge(type: "IN_CATEGORY", direction: OUT)
}

model Category {
  id    Int    @id @default(autoincrement())
  name  String
-  posts Post[] @relation(references: [id])
+  posts Post[] @edge(type: "IN_CATEGORY", direction: IN)
}

enum Role {
  USER
  ADMIN
}

Secondly, Relationship Properties
Relationships can have properties/fields, and this is a completely different concept from what anyone using Prisma is used to. To explain this concept I'm going to introduce you to a new data model because the current model itself doesn't need relationship properties.
Here we have a simple ‘Movies’ example, you will find this data model all over Neo4j’s docs and so I find it relevant to use here, below I represent it firstly using the proposed Prisma schema and then in an abstract form:
model Movie {
  title       String
  imdbRating  Int
  actors      Person[] @edge(type: "ACTED_IN", direction: IN)
}

model Person {
  name  String
  age   Int
}

We have already established that ‘relationships are first class’ in Neo4j and so that means they have mostly the same features as nodes. Given this model, what if we wanted to record what roles a given actor played in a movie? If we were using SQL for example we may make an intermediary ‘metadata’ table that contains the role information and facilitates the connection of the data. However, with relationship properties, this does not need to happen.

In this abstract model, you will see that now the relationship itself contains properties and so we need some way to model this in the Prisma schema. Below I propose my solution to this, it uses the already mentioned @edge attribute, where inside the new properties key each property is defined. You can think of a relationship property as a field on the relationship and with that, they should support all the same datatypes as those on nodes.
model Movie {
  title       String
  imdbRating  Int
  actors      Person[] @edge(type: "ACTED_IN", direction: IN, properties: { roles String[] })
}

model Person {
  name  String
  age   Int
}

Now that we have a way to represent properties on relationships we also now need to establish a way of reading and writing those properties. Given for example we use the Prisma client and the current way to read and write data:
await prisma.movie.findMany({
  where: { title: 'Forrest Gump' },
  select: {
    title: true,
    imdbRating: true,
    actors: {
      // What about the relationship properties?
      select: {
        name: true,
        age: true,
      },
    },
  },
})

await prisma.movie.create({
  data: {
    title: 'Forrest Gump',
    imdbRating: 8.6,
    actors: {
      create: [
        // What about the relationship properties?
        {
          name: 'Tom Hanks',
          age: 70,
        },
      ],
    },
  },
})

What about the relationship properties? Where do we interact with them?
Here I would like to point you to the Relay Connections Spec https://relay.dev/docs/guides/graphql-server-specification/#connections, where they have established a way to model relationships along with their metadata in GraphQL, and I believe that we could model that exactly in Prisma.  Given we use the relay spec, here is the above read and write using it:
await prisma.movie.findMany({
  where: { title: 'Forrest Gump' },
  select: {
    title: true,
    imdbRating: true,
    actors: {
      select: {
        edges: {
          roles: true,
          node: {
            name: true,
            age: true,
          },
        },
      },
    },
  },
})

await prisma.movie.create({
  data: {
    title: 'Forrest Gump',
    imdbRating: 8.6,
    actors: {
      create: [
        {
          edge: {
            roles: ['Forrest Gump'],
            node: {
              name: 'Tom Hanks',
              age: 70,
            },
          },
        },
      ],
    },
  },
})

One thought I would like to leave here that's relevant not just even in the case of a Neo4j provider. Should Prisma think of changing its API to support this model, as It opens up the ability to support more DB types, provides flexibility to paginate on nested relations/edges, and finally provides a platform to read and write any other metadata along with the relation?
Finally, Graph Capabilities
Graphs are very powerful, firstly you can use them in a general-purpose way just like any other DB you use but they come with some superpowers. The most spoken about superpower that graphs ship with is the ability to recommend data. Here we look at ways Neo4j has exposed this superpower via its GraphQL lib, and then leave room for others to bring ideas and suggestions to the table.
What are recommendations all about?
Given in Neo4j you want to query an actor and its movies you could write something like:
MATCH (tom:Person {name: 'Tom Hanks'})-[:ACTED_IN]->(movie:Movie)
RETURN actor {.*, movies: collect(movie)} as actor

This would return a JSON-like structure with each actor and its connected movies, however, what's really powerful is that you can use this same 'pattern matching' and path to grab for example 'co actors' traversing the graph to other actors:
MATCH (actor:Person {name: 'Tom Hanks'})-[:ACTED_IN]->(:Movie)<-[:ACTED_IN]-(coActor:Person)
RETURN actor {.*, coActors: collect(coActor)} as actor

Read more about recommendations here: https://neo4j.com/developer/cypher/guide-build-a-recommendation-engine/
The questions now are

How do we model this using Prisma schema?
How would you query it?
Is it something Prisma should support?

Looking at how Neo4j does it
The Neo4j lib https://github.com/neo4j/graphql shares some similarities with Prisma. On their side, you define a GraphQL schema with annotations about the underlying data model and then you get a generated GraphQL API. The methods on that GraphQL API are a very similar feature set to what Prisma exposes. With this particular problem, recommendations, they have an 'escape hatch 'the cypher directive'. Below I show you a simple GraphQL schema:
type Movie {
    title: String
    imdbRating: Int
    actors: [Person] @relationship(type: "ACTED_IN", direction: IN)
}

type Person {
    name: String
    age: Int
}

Then if you were to pop this into the Neo4j engine you could for example query for the actors and their movies:
query {
    actors {
        name
        age
        movies {
            title
            imdbRating
        }
    }
}

If you wanted to use the escape hatch and introduce recommendations into your schema you annotate the schema with the Cypher directive and write custom logic:
type Movie {
    title: String
    imdbRating: Int
    actors: [Person] @relationship(type: "ACTED_IN", direction: IN)
}

type Person {
    name: String
    age: Int
    coActors: [Person] @cypher(statement: """
        MATCH (this)-[:ACTED_IN]->(:Movie)<-[:ACTED_IN]-(coActor:Person)
        RETURN coActor
    """)
}

Then you can query the coActors this just like any other field where the predefined Cypher is injected into the generated query:
query {
    actors {
        name
        age
        coActors {
            name
            age
        }
    }
}

If Prisma were to adopt this approach, It could look very similar, given we have our already established data model, lets add the Cypher attribute in there:
model Movie {
  title       String
  imdbRating  Int
  actors      Person[] @edge(type: "ACTED_IN", direction: IN, properties: { roles String[] })
}

model Person {
  name  String
  age   Int
  coActors Person[] @cypher(statement: """
        MATCH (this)-[:ACTED_IN]->(:Movie)<-[:ACTED_IN]-(coActor:Person)
        RETURN coActor
    """)
}

Then, you could use PrismaClient to select that field:
await prisma.person.findMany({
  where: {
    name: 'Tom Hanks',
  },
  select: {
    coActors: {
      select: {
        name: true,
      },
    },
  },
})

This is a great approach that would allow users to cover all their complex use cases, however, listed below are some issues with this.
Problems with this approach

You can't tell what the user will write
It may not be the right level of abstraction for Prisma users
Brings the DB mechanics to the schema

Generating Reccomendations
Above we dived into the Cypher directive/attribute however could there be a possibility to pre-generate a generic enough set of recommendations? Given our schema do we have enough information to auto-generate the coActors field, and what would that look like?
Of course, the field would not be called coActors as that's a general construct we have given this particular graph traversal, however, could the API expose enough fields/input for users to specify a particular graph traversal?
Summary
As someone who is part of the Prisma team I would like to express that this comment represents no commitment from Prisma or any members of the Prisma team to support Neo4j. To add another connector requires an immense amount of engineering effort and there are soo many unknowns and gotchas with doing so that this comment doesn't do justice. I felt qualified to share my opinions and finding on this topic seeming that I previously came from working in the Neo4j team, on the GraphQL library and so I feel as though my understanding and learnings from doing so can, and should be shared here.

+1

+1

+1

Any update on this?

Hi there,
As part of our effort to clean up our GitHub repo, we’re closing this issue to help focus on features and improvements that align with our focus areas and those of our community.
I thank our former colleague Dan for his write-up above, but supporting Neo4j is currently out of our foreseeable plans.

If this is issue is still important to you, feel free to reopen the issue and get like-minded users to upvote as we will prioritize fixing issues based on community interest. If you do reopen the issue, please ensure that there’s a clear method for us to reproduce it.
For more insight into our priorities and how we’re thinking about the future of the Prisma ORM, please see our recent blog post: https://www.prisma.io/blog/prisma-orm-manifesto.
Thank you for your understanding and support!

The graph database community is in dire need of TypeScript-based data access layers. Unfortunately, Prisma won't be part of this, but given the significant differences in graph database implementations, I completely understand Prisma's decision not to integrate. RIP to the demand of this since Jan 16, 2018. May the Cypher(Now Open Cypher) find its place in the ecosystem somewhere 🙏

Hi there,
As part of our effort to clean up our GitHub repo, we’re closing this issue to help focus on features and improvements that align with our focus areas and those of our community.
I thank our former colleague Dan for his write-up above, but supporting Neo4j is currently out of our foreseeable plans.
If this is issue is still important to you, feel free to reopen the issue and get like-minded users to upvote as we will prioritize fixing issues based on community interest. If you do reopen the issue, please ensure that there’s a clear method for us to reproduce it.
For more insight into our priorities and how we’re thinking about the future of the Prisma ORM, please see our recent blog post: https://www.prisma.io/blog/prisma-orm-manifesto.
Thank you for your understanding and support!


[FEATURE] Upgrade Vega-tooltips plugin version

Is your feature request related to a problem?
The current version of the vega-tooltip package is 0.24.2 and doesn't support customizing the vega-tooltip.
What solution would you like?
Update the vega-tooltip version to 0.30.0 which does support customizing via formatTooltip API
https://github.com/vega/vega-tooltip/blob/next/docs/APIs.md#options

The update was made in the OSD - https://github.com/opensearch-project/OpenSearch-Dashboards/pull/3432


Increase visibility of this repo

Since it is preferred that some features live as an external package, it helps to be able to find them more easily. For that purpose, could you add the topics admin-on-rest, react-admin to this repo?
And could you add an examples paragraph to README.md that shows screenshots of how the UI of EmbeddedArrayField of EmbeddedArrayInput look like?

Done.


Saul 192

refactoring how keys are loaded on token restriction.
TODO: fix X509 certificate issue

pr ready to be reviewed


dump stlf"ed" data

@gleckler1 request we would like to save climatologies used once masked.

@gleckler1 thank you for pushing this, it revealed a big error in the way sftlf are autogenerated! So this will fix tht as well.
Now turns out what you're asking for is already "sort of" implemented. The filename_output_template right now is supposed to contain a "%(region)" keyword that is used to dump various region into separate files.
Are you ok with keeping this?
You're call, just let me know.
For now I'll assume we keep the code "as is" and will send a PR that fixes the autogen (new issue #224 ) if you're ok with code a sis just close this issue otherwise comment in here
Thanks again.

Global, land, ocean results look plausible.
Creating three separate climatologies, but no masking in any of them...

@doutriaux1 @durack1 need to confirm this one was resolved

@doutriaux1 Just confirmed results for land and ocean model data.   Not yet outputting interpolated obs, but thats another issue....


Use version 0.2 in guide overview

Codecov Report

Merging #114 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #114   +/-   ##
=======================================
  Coverage   80.85%   80.85%
=======================================
  Files          81       81
  Lines        4915     4915
=======================================
  Hits         3974     3974
  Misses        941      941

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a59e1dc...7c8e534. Read the comment docs.

Thank you!


Verify option to cli

Added an option for verifying that the generated types are up-to-date. This can be especially useful in CI/CD pipelines.

Published in<EMAIL_ADDRESS>


populate IP/port info for endpoints

Address https://github.com/Akshay-Venkatesh/ucx-py/issues/111
@mrocklin Can you lmk if this suffices? Note that I'm not populating local port number at client side of the endpoint. If the port isn't listening, I don't see how the port will be used by anyone.

Fixes https://github.com/Akshay-Venkatesh/ucx-py/issues/61
cc @TomAugspurger

Thanks for the review @TomAugspurger
Made some more changes to handle the case where the same process is the listener and client and added a test as well.

closing as we now have new python binding merged in: #180


fix: catalog pagination

What this PR changes/adds

Adds test cases to evaluate catalog pagination across multiple contract definitions.
Changes the logic which dynamically builds the offset and limit for each contract definition.

Why it does that
Previously the number of assets that should be skipped in the current iteration was dynamic but the limit was not, which caused the pagination to fail when across multiple contract definitions.
I decided to use a different approach that for me was easier to understand.
Based on the given range, the absolute offset and limit are static but could assume a different state on each iteration responding to the number of seen and fetched assets until that point. Therefore, I implemented the logic that sets the values of the dynamic offset and limit variables in the following expressions:

The dynamic offset must be equal to the absolute offset minus the number of assets seen, but never less than zero.
The dynamic limit must be equal to the absolute limit minus the number of assets already fetched, but never exceed the number of assets obtained in the current contract definition.

Linked Issue(s)
Closes #2493
Checklist

[x] added appropriate tests?
[x] performed checkstyle check locally?
[ ] added/updated copyright headers?
[ ] documented public classes/methods?
[ ] added/updated relevant documentation?
[x] assigned appropriate label? (exclude from changelog with label no-changelog)
[x] formatted title correctly? (take a look at the CONTRIBUTING and Etiquette for pull requests for details)

Codecov Report
Base: 64.70% // Head: 11.95% // Decreases project coverage by -52.75% :warning:

Coverage data is based on head (b54fad8) compared to base (3d0bd14).
Patch coverage: 0.00% of modified lines in pull request are covered.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
Additional details and impacted files
@@             Coverage Diff             @@
##             main    #2494       +/-   ##
===========================================
- Coverage   64.70%   11.95%   -52.75%
===========================================
  Files         846      846
  Lines       17533    17531        -2
  Branches     1066     1064        -2
===========================================
- Hits        11344     2096     -9248
- Misses       5754    15339     +9585
+ Partials      435       96      -339

Impacted Files
Coverage Δ

...ctor/contract/offer/ContractOfferResolverImpl.java
0.00% <0.00%> (-100.00%)
:arrow_down:

.../main/java/org/eclipse/edc/spi/result/Failure.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...main/java/org/eclipse/edc/spi/query/SortOrder.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...c/main/java/org/eclipse/edc/policy/model/Rule.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...n/java/org/eclipse/edc/spi/event/EventPayload.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...main/java/org/eclipse/edc/policy/model/Action.java
0.00% <0.00%> (-100.00%)
:arrow_down:

.../src/main/java/org/eclipse/edc/sql/DoorKeeper.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...in/java/org/eclipse/edc/policy/model/Operator.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...org/eclipse/edc/util/stream/PartitionIterator.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...va/org/eclipse/edc/spi/event/asset/AssetEvent.java
0.00% <0.00%> (-100.00%)
:arrow_down:

... and 526 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


applied StyleCI style fixes

Applies the recommended preset from StyleCI to clean up some stylistic stuff.

While I am a big fan of consistent code styling, I'm not 100% sold on this.  There are a handful of "definitive PHP code styles" out there it seems, including PSR-2, and several ways of enforcing them.
StyleCI seems pretty cool, but I'd rather it start out with some kind of configuration file that's checked into the project, rather than just fixes per merge request.  Something like what http://editorconfig.org/ provides.
And, no matter what anyone else says, tabs are always better than spaces for indenting IMHO.  ;)

That sounds like a good way to go. Thought I'd send the pr over anyway :) I agree with the tabs! I've been trying out spaces recently for PSR-2's sake. Feels strange...

If there was something that could be included in the project that:

was fairly standard (i.e. PSR-X),
could be tweaked (e.g. tabs instead of spaces),
could enforce the code-style in the major editors (i.e. Sublime Text, PHPStorm, Zend, Atom), and
wasn't super crazy to implement

I'd be all over that. ;)

FWIW, I bit the bullet and changed the code to PSR-2.  Feel free to look through and let me know if I missed anything (I mostly used PHPStorm's auto code-styling to convert everything).

Oh nice! I'm excited to take a look.


Support non-electing peers via --init-peer-count=0

Initial PR to provoke design discussion on the following points:

I have used a gossipState ring update message where both Ring and Paxos are nil as a way for non-electors to poll their peers for a ring - are we happy with that, or would a separate message type be preferable
Most of the changes to allocator.go (various alloc.paxos != nil protections, introduction of determinePaxos) could be reverted if paxos.Node became an interface with electing and non-electing variants. The latter would implement null versions of GossipState/Consensus/Propose/Update/Think and paxos.NewNode could then be made to return the correct implementation based on the value of quorum
--init-peer-count=0 isn't a great UI IMO; I would rather have something explicit like --disenfranchise that conflicts with the specification of --init-peer-count
weave status/weave report need to be updated; suggestions for how that should look are welcome

Thoughts @rade @dpw @bboreham?
Fixes #1743.

I am experimenting with a variant of (2) the involves some wrappers to avoid polluting the paxos package.

I am experimenting with a variant of (2) that involves the use of wrappers to avoid polluting the paxos package.

Why do you think there is pollution? The paxos package seems like a very natural place for Participant to live.

When an ip allocation happens on a non-elector, this should prompt the establishment of a ring among the electors, even though none of them themselves may have required an election.

This is now achieved by calling establishRing if we are polled by a non-elector but have no ring to send back. If we're an elector, we will propose, otherwise it's a no-op.

@rade simplified update as discussed - PTAL.
Will think about a smoke test - needs docs also.

Take #3.

I wonder about the scalability/performance implications of this. In a system of electors, there is a good chance that the first node on which an allocation happens will be the only one that makes a paxos proposal. It is only when there are near-simultaneous first allocations on multiple nodes that we get competing proposals. By contrast, if the first node on which an allocation happens is a non-elector, it will likely trigger proposals from all electors. This seems less than ideal.

Some thoughts:

Is this really a problem at all? In the worst case scenario (100 node elector set proposing simultaneously) how long does consensus take? I have been thinking about using terraform as a way to perform infrequent but more realistic scale testing than can be achieved with multiweave, perhaps testing this with AWS spot instances would be an interesting pilot
Our tentative set of use-cases suggest you're unlikely to be using non-electors in conjunction with a large set of electors, so even if the above takes a long time it's unlikely to be encountered in practice
Can be obviated entirely by weave consense in the elector set during initial cluster commissioning - was going to recommend that as a best practice anyway
The simplicity of the current implementation is compelling

The above notwithstanding, how could we improve the solution? We could make alloc.propose behave differently for observers:

Unicast a ring request to a random peer which is known to be an elector (this information is not disseminated at the moment)
Receiving elector proposes via broadcast to its quiescent peers; they accept the proposal, and form a ring
If the above happens within 5s, the next time the observer picks a random elector to talk to it'll probably have a ring available

If observers don't know which of its peers are electors, then either we have to live with the fact that we keep retrying every 5s until we hit one (not ideal) or alternatively we could have non-electors that receive ring requests forward them immediately to another random peer (vulnerable to message loops, also not ideal). Consequently I think we would need to add voter/observer status to the connection handshake to improve on what we have right now - is it worth it?

Is this really a problem at all?

I have no idea. Probably not at the 100 node scale.

Re point 1., I checked that down-level peers will tolerate a message with both ring and paxos nil.

Re point 1., I checked that down-level peers will tolerate a message with both ring and paxos nil.

Which point 1? Is this referring to backward compatibility?

Thanks @bboreham, I have squashed your commits in. Outstanding design questions:

How should weave status / weave report be updated
Choice of command line option --ipam-observer? --not-elector?

Once these are resolved I can amend the documentation accordingly.

Choice of command line option --ipam-observer? --not-elector?

I prefer non-ipam-specific terminology, on the grounds that we may well have other subsystems using the same mechanism one day.

Once these are resolved I can amend the documentation accordingly.

Will do this after the documentation restructuring has been merged.

Will do this after the documentation restructuring has been merged.

Please don't defer the documentation changes.

Which point 1?

This point 1:

I have used a gossipState ring update message where both Ring and Paxos are nil as a way for non-electors to poll their peers for a ring - are we happy with that, or would a separate message type be preferable

Is this referring to backward compatibility?

Yes, I was referring to backward compatibility, new peers talking to older peers.

--elector=false ?

--elector=false ?

The default can't be false, and having boolean options with a true default is exceptionally awkward.

I have chosen --observer. Added docs and rebased on master.


map in frames different version rotation (?)desync(?)

What is the output link of /viaversion dump?
https://gist.github.com/anonymous/79eb823884ec767e1351aedaae8ab3e2
Are you using any additional software like BungeeCord? If so, what software and version? (Please list your plugins as well)
Waterfall last build
as for plugins
only custom coded stuff and skinrestorer
How does this error happen? login? Using an item?
everytiem
Is there an error in the console? Use pastebin.com. Is there a kick message?
nope
https://imgur.com/a/9x6jO
1 screen made on 1.8.9 optifine client
2 screen made on 1.12 optifine client
I'm using https://github.com/m0bYczE/FramePicture for pic's however same problem with AnimatedFrames

Could you fill out all the information in the issue (specifically the "/viaversion dump")

@MylesIsCool
Ok. rdy

I can't reproduce this on a vanilla client.
I'm using MapManager, AnimatedFrames and tested on 1.12.1 client & 1.8.8 client (both show correctly).
Could you test with vanilla client without optifine?

yeah it is definately not a viaversion problem
caused by my modified spigot

Ah fair enough :), I'll close this issue as there's not much else I can help with in regards to it. I wish you luck in solving the issue.


Update Mixpanel.podspec

Reason: When integrating MixPanel 3.2.3 with Cocoapods, it fails, because the SDK refers iOS frameworks, that are not listed in the ios.frameworks section.
StoreKit referenced here: https://github.com/mixpanel/mixpanel-iphone/blob/master/Mixpanel/AutomaticEvents.h#L10
UserNotifications referenced here: https://github.com/mixpanel/mixpanel-iphone/blob/2b05839cbbdce7e87ab8d9ca8c0358cb90427e64/Mixpanel/Mixpanel.m#L17
Resolves: https://github.com/mixpanel/mixpanel-iphone/issues/728

Is TravisCI correctly configured? I see the following errors in the logs:
Unable to delete certificate matching "0950B6CD3D2F37EA246A1AAA20DFAADBD6FE1F75"security: AppleWWDRCA.cer: already in /Library/Keychains/System.keychain

$ openssl aes-256-cbc -k "$ENCRYPTION_SECRET" -in scripts/profile/HelloMixpanel_Ad_Hoc.mobileprovision.enc -d -a -out scripts/profile/HelloMixpanel_Ad_Hoc.mobileprovision
bad decrypt
1840:error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt:/BuildRoot/Library/Caches/com.apple.xbs/Sources/OpenSSL098/OpenSSL098-59.60.1/src/crypto/evp/evp_enc.c:330:
The command "openssl aes-256-cbc -k "$ENCRYPTION_SECRET" -in scripts/profile/HelloMixpanel_Ad_Hoc.mobileprovision.enc -d -a -out scripts/profile/HelloMixpanel_Ad_Hoc.mobileprovision" failed and exited with 1 during .

@patedit @pchien can you please advise reviewers for this PR? (Also do you know what is the reason for the TravisCI above?). Thank you very much for the help! :)

Sorry for the slow response @WiesnerPeti. There's a known issue with our travis config for external PRs. I'll merge these changes without the version bump - will change the version when releasing.


(v2) LoadClientStateMiddleware passes through a nil request

LoadClientStateMiddleware calls LoadClientState.
LoadClientState is returning a nil http.Request if either session state or cookie state is nil.   IMO, this function should never return a nil request.. that's almost certain to cause a nil pointer panic in further handlers.  This is what is happening to me.
https://github.com/volatiletech/authboss/blob/v2/client_state.go#L126
I'm not sure what the correct fix is... but returning a nil request without it being an error is definitely wrong here.

Yeah. That's seems totally insane. The way it should work is that if the state is not nil, it should modify r and allow the code to fall down to the next if statement to modify r more, the same thing happens there, and finally the original or modified r, is always returned at the end.
Having fixed this: https://github.com/volatiletech/authboss-clientstate/issues/2
I've found a new issue related in the same area. I believe the dropped error was intentional now, and that if session loading is failing for whatever reason (invalid session, expired timestamp (the error I'm getting atm), users can produce crazy inputs) we just ignore that error and return nil for the state because we don't care why we couldn't load it 99% of the time.
To log these errors seems perhaps overzealous. So the only thing to do is to revert the fix to client-state/2, and the API looks like: "If you can't load the session, it's not an error unless you couldn't load it for a show-stopper serious reason which means you'll never be able to load sessions again", or to log every session loading error that's passed back instead of panic'ing.
Currently the route I'm looking at is logging. Is it truly an error level log though? Tough calls.

It's possible that the proper fix for client-state should have been to switch on IsUsage and IsDecode errors, ignoring IsDecode.
http://www.gorillatoolkit.org/pkg/securecookie#Error

yeah, so this was originally hit because I forgot to set a cookie store key and a session store key... so they were being called with an empty []byte .... which evidently makes the clientstate return nil....

So I've fixed the issue that you've mentioned here in the exact way that I mentioned (because it was insane how it was happening before).
I've also touched up the error handling in the clientstate package to only return errors that are caused by non-decoding problems. That way your key missing error should still log problems and die on requests, but if John has a remember me cookie from a decade ago, it'll just be ignored.
Authboss rc6 has been tagged with the changes. Although as you know clientstate is a bit more of a wild-west repo, so you'll just have to fetch master of that :) Thanks for reporting all these issues.

Looks great, thanks for being so responsive.


Segmentation Fault error

Hi...
I'm having a problem trying to minify the html that I have in a database record. With other records are working fine.
It happens in an Amazon Linux 2 ec2 machine, but not in my ubuntu local machine.
You can find attached the html that is causing the error.
Thanks.
Regards.
segmentation_fault.txt
.

I'm unable to reproduce the issue on my machine either. If the input HTML document has a very deep tree (i.e. deeply nested tags), it may be possible that the segmentation fault is due to a too limited stack size limit, which may be more conservatively set on Amazon Linux 2. If the issue still persists, can you provide a core dump?


Update CI

Updates actions/checkout@v3 which gets rid of a warning message about Node versions

Combines the 3 separate build steps into one so they share the same build (quicker)

@patrickomatic
Thanks for the PR.
We will merge them.


Parser failing on !redo

Steps to reproduce the problem

create a submission issue or use https://github.com/qiskit-community/ecosystem/issues/21
type !redo in comments
go to https://github.com/qiskit-community/ecosystem/actions

What is the current behavior?
Failing on !redo

What is the expected behavior?
Should work on comment !redo.
Parser should work.

@mickahell ping :)

Logs https://github.com/qiskit-community/ecosystem/runs/3968809420?check_suite_focus=true

I created a new one and it works for creation and redo : https://github.com/qiskit-community/ecosystem/issues/59


[otbn] Fix register naming in BN.SID

grs1 is always the memory address, and always the second item in the
assembly (as in RISC-V). Fix the assembly syntax to match the
description.

This also requires ISS updates. I have those around locally, but want to verify some more things before I push it (and do so after the larger refactor from @rswarbrick is in).

@rswarbrick I'll have a fair amount of smaller (but hard to find) ISS fixes upcoming, the LID/SID instructions are currently broken anyways for different reasons. I think we could get this one in right now, under the assumption that the ISS is broken anyways.

Sounds good: let's merge this, then I'll fix up my proposed ISS change to match (and re-push that PR). I've also got a queue of ISS fixes, so I'd really appreciate getting that merged.


Some failing tests

What are the minimum necessary steps to reproduce this issue?

./configure
make
make check

What happens?
Some URI and Alert tests fails.
…
What were you expecting to happen?
Passing tests
Please paste any error or log messages here
$ make check
Making check in src
/Library/Developer/CommandLineTools/usr/bin/make  check-TESTS check-local
FAIL: test/test_axe
FAIL: qt/test/test_axe-qt
============================================================================
Testsuite summary for Axe Core 1.1.4
============================================================================
# TOTAL: 2
# PASS:  0
# SKIP:  0
# XFAIL: 0
# FAIL:  2
# XPASS: 0
# ERROR: 0
============================================================================
See src/test-suite.log
Please report to https://github.com/AXErunners/axe/issues
============================================================================
make[4]: *** [test-suite.log] Error 1
make[3]: *** [check-TESTS] Error 2
make[2]: *** [check-am] Error 2
make[1]: *** [check-recursive] Error 1
make: *** [check-recursive] Error 1

$ tail src/test-suite.log
PASS   : TrafficGraphDataTests::getRangeTests()
PASS   : TrafficGraphDataTests::switchRangeTests()
PASS   : TrafficGraphDataTests::clearTests()
PASS   : TrafficGraphDataTests::averageBandwidthTest()
PASS   : TrafficGraphDataTests::averageBandwidthEvery2EmptyTest()
PASS   : TrafficGraphDataTests::cleanupTestCase()
Totals: 9 passed, 0 failed, 0 skipped, 0 blacklisted, 6910ms
********* Finished testing of TrafficGraphDataTests *********
FAIL qt/test/test_axe-qt (exit status: 1)

(loki)(~/git/axe master %= )$ cat src/test-suite.log
========================================
   Axe Core 1.1.4: src/test-suite.log
========================================

# TOTAL: 2
# PASS:  0
# SKIP:  0
# XFAIL: 0
# FAIL:  2
# XPASS: 0
# ERROR: 0

.. contents:: :depth: 2

FAIL: test/test_axe
===================

Running 210 test cases...
test/alert_tests.cpp:167: error: in "Alert_tests/AlertApplies": check alert.CheckSignature(alertKey) has failed
test/alert_tests.cpp:167: error: in "Alert_tests/AlertApplies": check alert.CheckSignature(alertKey) has failed
test/alert_tests.cpp:167: error: in "Alert_tests/AlertApplies": check alert.CheckSignature(alertKey) has failed
test/alert_tests.cpp:167: error: in "Alert_tests/AlertApplies": check alert.CheckSignature(alertKey) has failed
test/alert_tests.cpp:167: error: in "Alert_tests/AlertApplies": check alert.CheckSignature(alertKey) has failed
test/alert_tests.cpp:167: error: in "Alert_tests/AlertApplies": check alert.CheckSignature(alertKey) has failed
test/alert_tests.cpp:167: error: in "Alert_tests/AlertApplies": check alert.CheckSignature(alertKey) has failed
test/alert_tests.cpp:167: error: in "Alert_tests/AlertApplies": check alert.CheckSignature(alertKey) has failed
test/alert_tests.cpp:173: error: in "Alert_tests/AlertApplies": check alerts[0].AppliesTo(1, "") has failed
test/alert_tests.cpp:174: error: in "Alert_tests/AlertApplies": check alerts[0].AppliesTo(999001, "") has failed
test/alert_tests.cpp:175: error: in "Alert_tests/AlertApplies": check alerts[0].AppliesTo(1, "/Satoshi:11.11.11/") has failed
test/alert_tests.cpp:177: error: in "Alert_tests/AlertApplies": check alerts[1].AppliesTo(1, "/Satoshi:0.1.0/") has failed
test/alert_tests.cpp:178: error: in "Alert_tests/AlertApplies": check alerts[1].AppliesTo(999001, "/Satoshi:0.1.0/") has failed
test/alert_tests.cpp:180: error: in "Alert_tests/AlertApplies": check alerts[2].AppliesTo(1, "/Satoshi:0.1.0/") has failed
test/alert_tests.cpp:181: error: in "Alert_tests/AlertApplies": check alerts[2].AppliesTo(1, "/Satoshi:0.2.0/") has failed
test/alert_tests.cpp:215: error: in "Alert_tests/AlertNotify": check r.size() == 4u has failed [0 != 4]
unknown location:0: fatal error: in "Alert_tests/AlertNotify": memory access violation at address: 0x00000000: no mapping at fault address
test/alert_tests.cpp:221: last checkpoint

*** 17 failures are detected in the test module "Axe Test Suite"
Database handles still open at environment close
Open database handle: unnamed/wallet.dat
FAIL test/test_axe (exit status: 201)

FAIL: qt/test/test_axe-qt
=========================

********* Start testing of URITests *********
Config: Using QtTest library 5.11.1, Qt 5.11.1 (x86_64-little_endian-lp64 shared (dynamic) release build; by Clang 9.1.0 (clang-9<IP_ADDRESS>) (Apple))
PASS   : URITests::initTestCase()
FAIL!  : URITests::uriTests() 'rv.address == QString("XwnLY9Tf7Zsef8gMGL2fhWA9ZmMjt4KPwg")' returned FALSE. ()
   Loc: [qt/test/uritests.cpp(55)]
PASS   : URITests::cleanupTestCase()
Totals: 2 passed, 1 failed, 0 skipped, 0 blacklisted, 2ms
********* Finished testing of URITests *********
********* Start testing of PaymentServerTests *********
Config: Using QtTest library 5.11.1, Qt 5.11.1 (x86_64-little_endian-lp64 shared (dynamic) release build; by Clang 9.1.0 (clang-9<IP_ADDRESS>) (Apple))
PASS   : PaymentServerTests::initTestCase()
QDEBUG : PaymentServerTests::paymentServerTests() PaymentServer::initNetManager: No active proxy server found.
QDEBUG : PaymentServerTests::paymentServerTests() PaymentServer::processPaymentRequest: Secure payment request from  "testmerchant.org"
QWARN  : PaymentServerTests::paymentServerTests() PaymentRequestPlus::getMerchant: Payment request: certificate expired or not yet active:  QSslCertificate("3", "03", "LxHILx+N3qwVoAcCmQ5cyw==", (), ("Expired Test Merchant"), QMap(), QDateTime(2013-02-23 21:26:43.000 UTC Qt::TimeSpec(UTC)), QDateTime(2013-02-24 21:26:43.000 UTC Qt::TimeSpec(UTC)))
QDEBUG : PaymentServerTests::paymentServerTests() PaymentServer::processPaymentRequest: Insecure payment request to  "PND1VDDpwyUj6HkApBjsXBCftkCwPHAoWk"
QWARN  : PaymentServerTests::paymentServerTests() PaymentRequestPlus::getMerchant: Payment request: certificate expired or not yet active:  QSslCertificate("3", "03", "LxHILx+N3qwVoAcCmQ5cyw==", (), ("Expired Test Merchant"), QMap(), QDateTime(2013-02-23 21:26:43.000 UTC Qt::TimeSpec(UTC)), QDateTime(2013-02-24 21:26:43.000 UTC Qt::TimeSpec(UTC)))
QDEBUG : PaymentServerTests::paymentServerTests() PaymentServer::processPaymentRequest: Secure payment request from  "testmerchant8.org"
QWARN  : PaymentServerTests::paymentServerTests() PaymentRequestPlus::getMerchant: Payment request: certificate expired or not yet active:  QSslCertificate("3", "06", "MiZaQ+g9lSHZGuHWkXZG+g==", (), ("Payment Request Intermediate 5"), QMap(), QDateTime(2013-02-23 22:59:51.000 UTC Qt::TimeSpec(UTC)), QDateTime(2013-02-24 22:59:51.000 UTC Qt::TimeSpec(UTC)))
QDEBUG : PaymentServerTests::paymentServerTests() PaymentServer::processPaymentRequest: Insecure payment request to  "PND1VDDpwyUj6HkApBjsXBCftkCwPHAoWk"
QWARN  : PaymentServerTests::paymentServerTests() PaymentRequestPlus::getMerchant: Payment request: certificate expired or not yet active:  QSslCertificate("3", "06", "MiZaQ+g9lSHZGuHWkXZG+g==", (), ("Payment Request Intermediate 5"), QMap(), QDateTime(2013-02-23 22:59:51.000 UTC Qt::TimeSpec(UTC)), QDateTime(2013-02-24 22:59:51.000 UTC Qt::TimeSpec(UTC)))
QWARN  : PaymentServerTests::paymentServerTests() PaymentRequestPlus::getMerchant: SSL error:  certificate signature failure
QDEBUG : PaymentServerTests::paymentServerTests() PaymentServer::processPaymentRequest: Insecure payment request to  "PND1VDDpwyUj6HkApBjsXBCftkCwPHAoWk"
QWARN  : PaymentServerTests::paymentServerTests() PaymentRequestPlus::getMerchant: SSL error:  certificate signature failure
QWARN  : PaymentServerTests::paymentServerTests() PaymentRequestPlus::getMerchant: SSL error:  unable to get local issuer certificate
QDEBUG : PaymentServerTests::paymentServerTests() PaymentServer::processPaymentRequest: Insecure payment request to  "PND1VDDpwyUj6HkApBjsXBCftkCwPHAoWk"
QWARN  : PaymentServerTests::paymentServerTests() PaymentRequestPlus::getMerchant: SSL error:  unable to get local issuer certificate
QWARN  : PaymentServerTests::paymentServerTests() "PaymentServer::verifyNetwork: Payment request network \"test\" doesn't match client network \"main\"."
QWARN  : PaymentServerTests::paymentServerTests() "PaymentServer::verifyExpired: Payment request expired \"1970-01-01 00:00:01\"."
QWARN  : PaymentServerTests::paymentServerTests() "PaymentServer::verifyExpired: Payment request expired \"1970-01-01 00:00:00\"."
QWARN  : PaymentServerTests::paymentServerTests() "PaymentServer::verifySize: Payment request too large (50001 bytes, allowed 50000 bytes)."
QWARN  : PaymentServerTests::paymentServerTests() "PaymentServer::verifyAmount: Payment request amount out of allowed range<PHONE_NUMBER>000000, allowed 0 -<PHONE_NUMBER>000000)."
PASS   : PaymentServerTests::paymentServerTests()
PASS   : PaymentServerTests::cleanupTestCase()
Totals: 3 passed, 0 failed, 0 skipped, 0 blacklisted, 901ms
********* Finished testing of PaymentServerTests *********
********* Start testing of CompatTests *********
Config: Using QtTest library 5.11.1, Qt 5.11.1 (x86_64-little_endian-lp64 shared (dynamic) release build; by Clang 9.1.0 (clang-9<IP_ADDRESS>) (Apple))
PASS   : CompatTests::initTestCase()
PASS   : CompatTests::bswapTests()
PASS   : CompatTests::cleanupTestCase()
Totals: 3 passed, 0 failed, 0 skipped, 0 blacklisted, 0ms
********* Finished testing of CompatTests *********
********* Start testing of TrafficGraphDataTests *********
Config: Using QtTest library 5.11.1, Qt 5.11.1 (x86_64-little_endian-lp64 shared (dynamic) release build; by Clang 9.1.0 (clang-9<IP_ADDRESS>) (Apple))
PASS   : TrafficGraphDataTests::initTestCase()
PASS   : TrafficGraphDataTests::simpleCurrentSampleQueueTests()
PASS   : TrafficGraphDataTests::accumulationCurrentSampleQueueTests()
PASS   : TrafficGraphDataTests::getRangeTests()
PASS   : TrafficGraphDataTests::switchRangeTests()
PASS   : TrafficGraphDataTests::clearTests()
PASS   : TrafficGraphDataTests::averageBandwidthTest()
PASS   : TrafficGraphDataTests::averageBandwidthEvery2EmptyTest()
PASS   : TrafficGraphDataTests::cleanupTestCase()
Totals: 9 passed, 0 failed, 0 skipped, 0 blacklisted, 6910ms
********* Finished testing of TrafficGraphDataTests *********
FAIL qt/test/test_axe-qt (exit status: 1)

Any other details
Master branch at 1b9d1f16a

this should be closed with #88 soon


Adding Subcategories

The admin should be able to add subcategories

Subcategories are added almost the same way as Categories.
The difference is:

parent=parentid

For more info on adding Categories check: Issue 13.


nix-build fails when "src" contains Unicode characters, i.e. "src =  /tmp/créatúr"

I have a couple of accented characters in my .cabal file (but only in the description, copyright and author fields). This file works fine when I build it on a non-NixOS system with sandboxes, but on a NixOS system, the nix-build step causes an error:
error: invalid character ‘�’ in name ‘créatúr’
(use ‘--show-trace’ to show detailed location information)

And if I do nix-build --show-trace, I get:
error: while evaluating the attribute ‘src’ of the derivation ‘haskell-creatur-5.9.6’ at "/nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/pkgs/development/haskell-modules/generic-builder.nix":123:3:
invalid character ‘�’ in name ‘créatúr’

I think this is a bug. Although the call to nix-build is what triggers the error, if I understand correctly, nix-build doesn't know anything about cabal files, so I suspect the problem occurs elsewhere.
The files creatur.cabal, shell.nix, project.nix, and default.nix are here: https://gist.github.com/mhwombat/716cd1cc181bac64991b'
My workflow:

I created default.nix.
cabal2nix ./. --shell > shell.nix
cabal2nix ./. > project.nix
nix-shell
nix-build (which produces the error)

In your workflow you run nix-build within a nix-shell? (Looks strange.)

I exit nix-shell before running nix-build. (I just use nix-shell to try things out). Sorry for the confusion.

I think the problem may be that the directory name itself contains accented characters ("créatúr"). Renaming the directory seems to have got me past that point.

nix-build can't cope with the Unicode characters in src. This is a bug in Nix, IMHO.

(triage) I supposed this hasn’t changed yet?

Can't reproduce this with src anymore:
$ cat default.nix
{ pkgs ? import <nixpkgs> {} }:

with pkgs;

stdenv.mkDerivation {
  name = "creatur";
  src = "/tmp/créatúr";
}
$ nix-build
these derivations will be built:
  /nix/store/kaz4iwkk2ng7ml42b6qy1vb64b1irn3x-creatur.drv
building path(s) ‘/nix/store/jjcvrmjk7q47iyzkgzw21ncmk53zv7bk-creatur’
unpacking sources
unpacking source archive /tmp/créatúr
do not know how to unpack source archive /tmp/créatúr
builder for ‘/nix/store/kaz4iwkk2ng7ml42b6qy1vb64b1irn3x-creatur.drv’ failed with exit code 1
error: build of ‘/nix/store/kaz4iwkk2ng7ml42b6qy1vb64b1irn3x-creatur.drv’ failed

However, name should still be Nix store compatible, and that means no Unicode characters. That's intended:
$ cat default.nix
{ pkgs ? import <nixpkgs> {} }:

with pkgs;

stdenv.mkDerivation {
  name = "créatúr";
  src = "/tmp/créatúr";
}
$ nix-build
error: invalid character ‘�’ in name ‘créatúr’


Fix for yarn 2.4.3

After #617 was merged our inventory automation ran and the PR to update the yarn inventory with 2.4.3 (#620) had failing tests that showed that the layout for this version did not match our other distributions.
This PR handles this edge case by renaming the bin/yarn.js script to bin/yarn which matches the binary we expect to find. This is slightly different the @yarnpkg/cli-dist layout which uses a portable shell script at yarn/bin that calls node /path/to/yarn.js.

While this fix does address the issue with using 2.4.3 as a global yarn binary, instead of having to add this special case that executes on every build, we've decided to exclude 2.4.3 from our inventory listing.
This should not impact applications that use 2.4.3 because in Yarn 2+ you are expected to commit the .yarn/release/yarn-vX.Y.Z.cjs file which controls the actual version of Yarn used during a build.  The globally installed Yarn simply operates as a wrapper to invoking the commited release version and that should be compatible with all of our other versions of Yarn in our inventory.


Update 6502 flags handling.

Some instructions did not set the appropriate flags and some others did set flags where they shouldn't.
I checked this against a few datasheets from 6502.org and they all seem to agree.

These look very reasonable. I'll review them shortly. Thanks @agatti


Cluster deploy on LInux failed

Executing hack/deploy on Ubuntu, i got the swarm dns issue we got previously:

most of the time, not always, kibana don't find elasticsearch (which is well there)
if kibana is ok, then agents don't find nats
then we got the message : "final service replicat smoke test failed exit 1"
and the cluster stops

looking logs and going into the containers, kibana or agent don't ping Elasticsearch or Nats respectively

is Swarm initialized on your Linux? If not, what is the result of sudo lsmod | grep ip_vs?

using command sudo modprobe ip_vsthe issue is solved, closing the issue


Connection left in bad state after a timeout

I have an issue where my connection is left in a bad state after a timeout or a very slow answer.
Here's a paste of my log, with log level set to 63. To explain a little what you see;
One "query" is really two queries, first one to mymeta (called the meta query from now on), and given the response from that one to mydata.response (called the data query from now on). The expected result from mymeta is {string, int}, and expected result from mydata is {timestamp, float}
All the queries below are the same queries over and over.
The server is an MSSQL 2012
The names and conditions are fake, but the log is copied from real executions. There's nothing fancy going on in the real query.
First query. Meta query already executed with no problem. Log left out as this is long as it is:
2015/08/19 10:06:59 Querying data from hour
2015/08/19 10:06:59 SELECT ts, col FROM mydata.hour WHERE id = @p1 AND ts BETWEEN @p2 AND @p3
2015/08/19 10:06:59 	@p1	10305
2015/08/19 10:06:59 	@p2	2015-07-11 00:00:00 +0000 UTC
2015/08/19 10:06:59 	@p3	2015-08-12 00:00:00 +0000 UTC
2015/08/19 10:07:29 SELECT ts, col FROM mydata.hour WHERE id = @p1 AND ts BETWEEN @p2 AND @p3
2015/08/19 10:07:29 	@p1	10305
2015/08/19 10:07:29 	@p2	2015-07-11 00:00:00 +0000 UTC
2015/08/19 10:07:29 	@p3	2015-08-12 00:00:00 +0000 UTC
2015/08/19 10:08:25 Error when fetching data: read tcp mymssqlserver:1433: i/o timeout

Second query. Fails on meta query:
2015/08/19 10:08:53 SELECT f1,f2 FROM mymeta WHERE conditions = 1
2015/08/19 10:08:53 Error when fetching data: Error while finding finest aggregation level: Error when finding finest agg: mssql.StreamError=Invalid TDS stream: Unknown token type: 122

Third query. Fails on meta query:
2015/08/19 10:10:33 SELECT f1,f2 FROM mymeta WHERE conditions = 1
2015/08/19 10:10:33 Error when fetching data: Error while finding finest aggregation level: Error when finding finest agg: mssql.StreamError=Invalid TDS stream: Unknown token type: 101

Forth query. Succeeds with meta query. Notice how the raw results here are the expected results from a meta query, but the query was actually a data query:
2015/08/19 10:19:38 Querying data from hour
2015/08/19 10:19:38 SELECT dt, S4517B FROM mydata.hour WHERE id = @p1 AND ts BETWEEN @p2 AND @p3
2015/08/19 10:19:38 	@p1	10305
2015/08/19 10:19:38 	@p2	2015-07-11 00:00:00 +0000 UTC
2015/08/19 10:19:38 	@p3	2015-08-12 00:00:00 +0000 UTC
2015/08/19 10:19:38 (3 row(s) affected)
2015/08/19 10:19:38 Problem while scanning query result: sql: Scan error on column index 0: unsupported driver -> Scan pair: string -> *time.Time
[]string{"15min", "15"}
2015/08/19 10:19:38 Problem while scanning query result: sql: Scan error on column index 0: unsupported driver -> Scan pair: string -> *time.Time
[]string{"day", "<nil>"}
2015/08/19 10:19:38 Problem while scanning query result: sql: Scan error on column index 0: unsupported driver -> Scan pair: string -> *time.Time
[]string{"hour", "40"}

Can someone please tell me if I am doing something wrong, or if this is a bug in the library?

Please, ensure that Close is called after each query (see the documentation http://golang.org/pkg/database/sql/#Rows.Close)

It is. After every Query, there is a defer rows.Close()

With defer Close will be executed at the end of the function. If you have the queries in the same function Close is not called between queries. Call rows.Close() without defer after each query.  (There is a great resource for go sql interface http://go-database-sql.org/retrieving.html)

I updated my comment to cover that case while you wrote your answer. Sorry!
Anyway, my code effectively follows the example given in http://golang.org/pkg/database/sql/#DB.Query
No queries while having an active query, no recursive calls, always defer a Close().
My issue only arise the query after a query times out.

The connection times out from the client side but the server keep sending data.
We need to cancel the query on timeout (see #29)

I see. So there is nothing I can do about it on the client side (part from setting a high timeout value) until #29 is fixed?

Yes, setting a higher timeout (default timeout is 30 secs) is a good workaround until the bug is fixed.

Same (?) error showed up in some logs yesterday:
read tcp <IP_ADDRESS>:56764-><IP_ADDRESS>:1433: i/o timeout
sql: no rows in result set
mssql: The server failed to resume the transaction. Desc:d700000018.
mssql: New request is not allowed to start because it should come with valid transaction descriptor.
sql: no rows in result set
The server failed to resume the transaction. Desc:d700000020.
sql: expected 3 destination arguments in Scan, not 1
sql: expected 1 destination arguments in Scan, not 3
sql: expected 3 destination arguments in Scan, not 1
sql: expected 1 destination arguments in Scan, not 3

From that point on, those two sql: expected... errors repeat up until I restarted the application, at which point everything went back to normal. The end users found that if they clicked submit a few times it would eventually work, about every three clicks.
In my code I'm only calling scan twice for each request, once with three values and once with one. The code is pretty simple, and I've got unit tests verifying it works. So it looks like... they get switched around? Or something...

This was closed, but I'm experiencing the same error using the latest version; read timeouts and then failed to resume transaction and then scan errors that appear to just be out of order. It's going to be hard to repro but I'll try.

try again on latest version, it sets TCP keep alive to 30 seconds by default now


Black and Purple Textures

Some Textures are replaced with Black and Purple Textures. This happens with older resource packs but the textures work fine with Optifine.
Version: 1.20.1

Not mearly enough information to work with here. I'll just assume it's the texture atlas thing with 1.20 and you'll just have to update the packs.


Bug in RangeBar.onTouchEvent

Hello, please fix bug in RangeBar.onTouchEvent, line #433.
Current version:
case MotionEvent.ACTION_UP:
            case MotionEvent.ACTION_CANCEL:
                this.getParent().requestDisallowInterceptTouchEvent(false);
                onActionUp(event.getX(), event.getY());
                return true;

Replace with:
            case MotionEvent.ACTION_UP:
                this.getParent().requestDisallowInterceptTouchEvent(false);
                onActionUp(event.getX(), event.getY());
                return true;
            case MotionEvent.ACTION_CANCEL:
                this.getParent().requestDisallowInterceptTouchEvent(false);
                return false;

These changes fix scrolling issue (unprompted rangebar changes) when RangebBar is in scrollable view.

Thanks. Do you want to do a PR?

Unfortunately no, because i didn't make fork, it was some experiments.

Done, https://github.com/oli107/material-range-bar/pull/16


Consider built-in support for editorconfig

I've seen some mention of an Editor: indentation settings command but I can not seem to get to it (using the latest alpha 0.5.4).
It would be nice if LightTable would have builtin support for editorconfig:
http://www.editorconfig.org
which is a great way to create shareable editor settings across many editors and allows a project maintainer to dictate indentation settings for a repository.

:+1: it's kind of a good thing to enforce code style across the team.. So it's not so much about supporting editorconfig, but about supporting teams. That's the really important aspect of it. However editorconfig might be best of breed.

This is already in the Feature wishlist wiki page. If someone is interested in working on this, create a separate repo for the code – which should be an LT plugin.


flutter 1.9.1.hotfix.6 Gradle error

I am try to run my first app on flutter but i got this error.
Launching lib\main.dart on Android SDK built for x86 in debug mode...
Initializing gradle...
Resolving dependencies...
* Error running Gradle:
ProcessException: Process "C:\Users\ZESTARLAB\Documents\fasttrack\android\gradlew.bat" exited abnormally:
Starting a Gradle Daemon, 4 busy Daemons could not be reused, use --status for details

FAILURE: Build failed with an exception.

* What went wrong:
Could not create service of type ScriptPluginFactory using BuildScopeServices.createScriptPluginFactory().
> Could not create service of type ResourceSnapshotterCacheService using GradleUserHomeScopeServices.createResourceSnapshotterCacheService().

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org
Finished with error: Please review your Gradle project setup in the android/ folder.

BUILD FAILED in 2m 32s
  Command: C:\Users\ZESTARLAB\Documents\fasttrack\android\gradlew.bat app:properties

how could i resolve this?

Hi @adetoroe787
can you please provide your updated  flutter doctor -v
and your flutter run --verbose?
Also, to better address the issue, would be helpful
if you could post  a minimal code sample
or the steps to reproduce the problem.
Thank you

Without additional information, we are unfortunately not sure how to resolve this issue. We are therefore reluctantly going to close this bug for now. Please don't hesitate to comment on the bug if you have any more information for us; we will reopen it right away!
Thanks for your contribution.
Could everyone who still has this problem please file a new issue with the exact descriptions what happens, logs and the output of 'flutter doctor -v'  please.
All system setups can be slightly different so its always better to open new issues and reference related issues.


local variable ensembl_no referenced before assignment

File "/ifs/apps/apps/python-2.7.9/lib/python2.7/site-packages/ruffus/task.py", line 743, in run_pooled_job_without_exceptions
                                                         return_value =  job_wrapper(param, user_defined_work_func, register_cleanup, touch_files_only)
                                                       File "/ifs/apps/apps/python-2.7.9/lib/python2.7/site-packages/ruffus/task.py", line 541, in job_wrapper_io_files
                                                         ret_val = user_defined_work_func(*param)
                                                       File "/ifs/mirror/jenkins/CGATPipelines/CGATPipelines/pipeline_mapping.py", line 363, in buildReferenceGeneSet
                                                         if ensembl_no >= 78:
                                                     UnboundLocalError: local variable 'ensembl_no' referenced before assignment

Hi @TomSmithCGAT , Please fix.
Also, do not work with ENSEMBL version numbers. Instead test if the attributes you need are present and if not fall back to something else. And respect the boundaries between pipelines - do not try to guess what another pipeline has done, but only work with the data exported.
Best wishes,
Andreas

@AndreasHeger Thanks for the advice. The issue is that "source" column of the GTF file (column 2) no longer contains the expected values, e.g "processed_transcript" but now contains the actual source of the annotation, e.g "havana". The "processed_transcript" string is now contained in the attributes as a gene biotype which is lost during the creation of the geneset_all.gtf.gz file.
This can either be rectified in the annotations pipeline so that all final gtfs are as expected regardless of ensembl version number, or rectified downstream where the difference in ensembl gtf causes a problem. It makes more sense to me to modify pipeline_annotations. What do you think?

Hi @TomSmithCGAT , the purpose of pipeline_annotations is to provide genomic annotations in a standardized format - so yes, dealing with the effects of different ENSEMBL versions needs to be implemented in pipeline_annotations. The correct way is not to patch, but to generalize the current implementation such that both old and new ENSEMBL can be accommodated.
Best wishes,
Andreas

@AndreasHeger - OK, I'll try and make pipeline_annotiations consistent between ensembl versions.

Thanks! Happy to help.

Issue resolved by modifying GTF.Entry class and gtf2gtf.py to retain gene_biotype tag in attributes of merged GTFs. Removed patch from pipeline_mapping.py and PipelineMapping.py.
Changes made on following branches:
TS-pipeline_annotations_constistency_ENSEMBL_version (CGATPipelines)
TS-Add_re-source_method_to_gff2gff (cgat)

Thanks, this has been fixed.


Add info about faucet's required pool id format - bech32

When using faucet to delegate funds to pool and entering pool ID in hex format - 3ad683d9d9f38f43c1b638f0e79f1c91173d324c0c3055971aacf575 there is an error.
Request Method: GET
Status Code:400

Request URL:
https://faucet.preview.world.dev.cardano.org/delegate?poolid=3ad683d9d9f38f43c1b638f0e79f1c91173d324c0c3055971aacf575&api_key=&g-recaptcha-response=03AL8dmw-05XFCq0ZZlmR9qg_mIVsjEXrNlxC3VmSAgy8BgEPKXOzrC6tQLaRclR8qPlzfuDXveWykFGJwMG36ogKt0dvmYoYOgClUzsmLR-XU-W6KHDBvXywLgEzZDjkMYCKBADt7_Ddg-VjdKvgliZA26Hs-XhQZ75Y3S29NAY51dhNV6hxZPcTh0r-JxfwxAyNa--1cU31vhrZAdAxhVi9xMCyASbr3lC82XYQRYkDezI1-HRxWqJmHFDE34lzCI6hDpiP7NK0bbi8Yebo5YQDDyfSTA_LFYY1QHYyrGJIh-CUDGEj8JdeVJpnYQMoejWQhUm5Cq7zQkOMBuR1in_APnc7uaySXXyV7F8P5IIYfqYX-eQECE7aRa-HpOWa-wOErKlmMQwGSGrXnXV-kAQi512XZ3qom76ZexMk17ud42c0ofeX5l2XBEOYdjSNFEjYF-KNSuSoaiU4fDPcZdLDmW3jUGD6x0ylLj9HnOQg9NQFR-3eMSjinN6wLyLhWnCoJnB-TRsohe5LkoJin_lMJvMD9QEsbBF9NkAXv-U1KxIr69wAAUmQ

This error however is not present when using bech32 format for pool ID - e.g. pool18ttg8kwe7w858sdk8rcw08cujytn6vjvpsc9t9c64n6h27wza2u.
I ran into this issue when trying to delegate funds to my test pool.
This PR updates documentation and aims to avoid confusion.
There is an enhancement ticket that is supposed to fix it and support two formats in the future: https://github.com/input-output-hk/cardano-world/issues/95
Original slack thread here: https://input-output-rnd.slack.com/archives/C03NYMJBD7F/p1687265266352119

@CarlosLopezDeLara has approved, that is good enough for me.


Add accessibility modifier info to interfaces

Addresses #5261

@rpetrusha Nope, I think it's fine as-is. Everything else is formatted this way.


Add information about defer & compile

What are you changing in this pull request and why?
Adding information about how defer and compile work together.
Stumbled upon this functionality and decided to document it for the benefit of others.

hey @siljamardla , thanks for opening this up! i'm going to work with the team internally with this and will revert back with more info! really appreciate your contributions ✨ !

hey @siljamardla , can you provide more insight on your proposed changes?  Wondering if you're experiencing this scenario where you have dev artifacts that dbt is favoring?

hey @siljamardla , can you provide more insight on your proposed changes? Wondering if you're experiencing this scenario where you have dev artifacts that dbt is favoring?

Will have to test again to be sure. But if you have tested and you're sure that defer also works with full compile, feel free to modify my PR. The main point is to make it known that defer also works with compile.

Hey
Tested again.

I dropped my dev schema
I ran dbt compile --defer --state target/live (this is where I store the latest live manifest)
Checked the compiled output. All the refs use my personal schema name.
I recreated my dev schema. I checked that there are no tables in it.
I ran dbt compile --defer --state target/live again
Checked the compiled output, again all the refs use my personal schema name.
I ran dbt compile --defer --state target/live --select my_model_name > target/tmp_my_model_name.sql
Checked the output, in this one all the refs have been compiled to use the live schema names
Just in case, I deleted my target folder and went through the same routine again, still the same outcome

Thus, I stand by the claim that defer does not work on dbt compile, but it does work when you compile something specific.

@mirnawong1, even if you plan to investigate and/or fix anything, I think it's valuable information that should be added to the docs meanwhile.

Thanks so much @siljamardla ! Will take a look at this tomorrow and also would like to cc @dave-connors-3 as well!

@mirnawong1 @siljamardla hello there! i definitely believe this to be unintended behavior, and should go through core support, and not land in the docs site until we get a bit more clarity into what's going on!
@siljamardla, would you be willing to open up an issue on dbt-core, and document the steps you showed above?

hey @siljamardla , thank for your patience here! I've had a chat with @dave-connors-3 and this sounds like unintended behavior and possibly a bug. could you please open a core issue for the team to look at?
appreciate your insight and looking forward to your future docs contributions ✨!  thank you as well @dave-connors-3 !

i'll close this pr out and happy to reopen at a later point. thanks again!

As suggested, I've created an issue:
https://github.com/dbt-labs/dbt-core/issues/9991


subfield data element management

Hi Igor,
It's possible to manage subfields as tag for some data element?
For exemple, field 55 contain many subfields with specific length and format.
FLD           | LENTGH    | VALUE
FLD (055) : (101) :                                                                       .
	> TAG (82  ) ....: [3C00]                                                             .
	> TAG (9F33) ....: [E0F8C8]                                                           .
	> TAG (95  ) ....: [0080008000]

Kindly assist

Hi!
What do you mean by "managing"? A callback function?

Hi,
Something like way to define each subfields as you implemented for ISO fields fields before packing the iso message .
Cause according to somes host specifications for VISA or others there as some subfields by each fields like fields 48, fields 55 etc...

Well, made an improvement.
Two new methods to mind:

Before the whole packet is packed: https://github.com/juks/iso-8583-socket-queue/blob/0ceaa81efcce51536f41a1b4cf12702f21e3a345/lib/iso8583/lib/packager/smartVista.js#L440
Before each field is packed: https://github.com/juks/iso-8583-socket-queue/blob/0ceaa81efcce51536f41a1b4cf12702f21e3a345/lib/iso8583/lib/packager/smartVista.js#L114

Hi igor.
thanks for your feedback.
I'm afraid to not understand your implementation.
Kindly provide an use case for exploitation. I mean an example for field 55

You should do like so for the field definition:
  '55': {
    length: 255,
    name:   'EMV Data',
    type:   'll-bin-char',
    alias:  '',
    beforePack: function(fieldValue) {
        // Work with tag (TLV) values here
        return fieldValue;
    }
  },

If your message has no field 55 and you want to add it explicitly, add it into the beforeBack method:
  beforePack: function(data) {
    data['55'] = Buffer.from('9f2608571f1e10d4fa4aac9f2701809f100706010a03a4b8029f37045bb074729f3602000c950500800010009a031508189c01009f02060000000010005f2a02064382023c009f1a0206439f03060000000000009f3303e0f0c89f34034403029f3501229f1e0835313230323831358407a00000000310109f41030000565f340101', 'hex');
    return data;
  }

Thank for your revert.
Something like this i it correct?
  '55': {
    length: 255,
    name:   'EMV Data',
    type:   'll-bin-char',
    alias:  '',
    beforePack: function(fieldValue) {
       '9F26': {
    length: 8,
    name:   'Application cryptogram',
    type:   'fixed-b',
    alias:  ''
  }
        return fieldValue;
    }
  },

Unfortunatelly, there is no tag layer abstraction inside the beforePack right now. So what you get there is entire field 55 value, that you need to break into parts and process on your own.

Ok i see
So what do you mean by

// Work with tags (TLV) values here

Some  example

I mean you have fieldValue variable there, representing the field 55 value. So to alter the EMV tags you need to implement it in this method.

@scalpovich you could implement this:
https://www.npmjs.com/package/node-tlv

@cavebring thanks for the information.
I will take a look quickly


MC non triggered events

I was trying to see if I could use ctapipe to read in sim_telarray data and get both the triggered data and all the simulated MC data. This would be used to calculate, for example, the trigger efficiency with respect to photoelectrons in the image. As far as I can see this isn't possible at the moment?
Looking in hessio_event_source(...) in ctapipe/io/hessio.py
with open_hessio(url) as pyhessio:
    # the container is initialized once, and data is replaced within
    # it after each yield
    Provenance().add_input_file(url, role='dl0.sub.evt')
    counter = 0
    eventstream = pyhessio.move_to_next_event()
    ...

I see that there could be an option to somehow switch to eventstream = pyhessio.move_to_next_mc_event() which would then provide you with all the simulated data. When testing this however, I came across something I don't quite understand. If I run some example code such as:
source = hessio_event_source(file_path, max_events=None)
for event in source:
    print(event.r0.event_id, event.mc.energy, sum(event.mc.tel[1].photo_electron_image))

With the original code [move_to_next_event()]
1 0.7434797286987305 TeV 141
2 0.5755431056022644 TeV 223
3 0.32009783387184143 TeV 81
4 0.7020784020423889 TeV 218
5 0.2921573519706726 TeV 89
6 2.6281118392944336 TeV 1149
7 4.678738117218018 TeV 2734
8 0.3247097432613373 TeV 123
9 0.937703549861908 TeV 139
10 4.912286281585693 TeV 183
11 1.148128628730774 TeV 124
12 1.5528963804244995 TeV 359
13 0.3332664668560028 TeV 189
14 10.62702465057373 TeV 300
15 0.4885904788970947 TeV 227
16 14.384729385375977 TeV 925
17 0.42923223972320557 TeV 110
18 1.918776273727417 TeV 202
19 3.072026252746582 TeV 202
20 1.6568242311477661 TeV 172

and with edited code [move_to_next_mc_event()]
100 0.21774375438690186 TeV 0
200 0.326642781496048 TeV 0
300 0.7434797286987305 TeV 0
400 0.46060770750045776 TeV 141
500 0.4362963140010834 TeV 31
600 0.28762349486351013 TeV 31
700 0.5887979865074158 TeV 31
800 0.5755431056022644 TeV 28
900 0.4464867115020752 TeV 223
1000 0.42257633805274963 TeV 223
1100 0.42465123534202576 TeV 223
1200 0.2719481289386749 TeV 223
1300 0.9962665438652039 TeV 223
1400 2.7529940605163574 TeV 45
1500 0.23600557446479797 TeV 22
1600 0.39889732003211975 TeV 22
1700 0.4632396399974823 TeV 22
1800 1.089866042137146 TeV 63
1900 0.2184465080499649 TeV 15
2000 0.2836531102657318 TeV 15

If you take the event which has a simulated energy of 0.7434797286987305 TeV, this has event number 4 and a total of 218 p.e. in the image in the first case, and event number 300 and a total of 0 p.e. in the second case (where the correct p.e. value seems to be associated with the next event).
Additionally if I try to calculate the trigger efficiency, I can only obtain a sensible answer if I remove all the duplicated p.e. values.
This could be a non issue if there is a different way to read in the simulated data?

Hi Tom,
FACT have this bit of code that might help called pyeventio (https://github.com/fact-project/pyeventio), you could theoretically read out the MC energies from the corsika.gz file you ran simtelarray with. Perhaps it'd be worth integrating this into ctapipe?

We hadn't considered supporting non-triggered MC events, so there is probably a logic error when simply changing to the move_to_next_mc_event method when filling the containers. It's never been tested.   My guess is it's something simple that we could fix however.  If you would like to commit this and debug it, please open a pull-request.
You can also in principle get a histogram of the simulated events from the MC file, but we don't yet have an accessor method in pyhessio to get histograms (there is an open issue about that: cta-observatory/pyhessio#49 ).

pyeventio now fully supports simtel files, you can should be able to get every information you need.
We also moved pyeventio to cta-observatory and you can pip install eventio.
To get information about all simulated events, loop through a file and look at all MCShower and MCEvent.
There is one MCShower per simulated shower and then for each shower there are number of reuse MCEvent which basically just contain the impact position.
See this example: https://github.com/cta-observatory/pyeventio/blob/master/examples/iter_mc_events.py


docker attach not exiting when container exits

Very simple test:
docker attach $(docker run -tid busybox sh -c 'sleep 2; echo exiting')

The command will show the "exiting" message, but never exits. The container is in fact dead, as doing a docker ps shows it gone. However pressing any key will then cause the attach to exit.
# docker version
Client version: 1.4.1
Client API version: 1.16
Go version (client): go1.3
Git commit (client): 5bc2ff8
OS/Arch (client): linux/amd64
Server version: 1.4.1
Server API version: 1.16
Go version (server): go1.3
Git commit (server): 5bc2ff8

# docker info
Containers: 74
Images: 47
Storage Driver: btrfs
Execution Driver: native-0.2
Kernel Version: 3.18.1-gentoo
Operating System: Gentoo/Linux
CPUs: 4
Total Memory: 7.512 GiB
Name: gadget
ID: KLI3:BPWQ:CK45:NJCL:ULCH:BV6Z:XDFM:YKTZ:EZ7D:UBE7:ZQCK:HOGN
Username: phemmer
Registry: [https://index.docker.io/v1/]

Can this be closed as a non-issue?

@acbodine Is this non-issue?

It appears to be waiting on the io.Copy here:
https://github.com/docker/docker/blob/master/api/client/hijack.go#L203

In your example docker attach $(docker run -tid busybox sh -c 'sleep 2; echo exiting') you are explicitly keeping your container's stdin stream open with the -i flag to docker run, and excluding the wrapping docker attach, your newly created container will sleep for 2 seconds and then print 'exiting' to it's stdout.
By looking through the source, when you run a docker attach the cli first queries the daemon for the json config of the container you are attempting to attach to. By looking at whether or not the container was created with a setting to keep stdin open or not (i.e. -i flag to docker run) the cli will either setup a client-side pipe to the container's stdin or it will not. If it does set up the client-side pipe then there are go routines in the daemon that are stuck waiting for input from the cli to forward on to the container you are attaching to. Hence why the cli is hanging and when you press a key, it will return to a normal state.
Honestly I'm still not sure if this is an issue. I did find that if you add the --no-stdin=true flag to your docker attach the cli will behave as you expected by explicitly not opening up the client-side pipe to the container's stdin regardless of the container config. Otherwise by removing the -i flag from your docker run - as @LK4D4 has mentioned - the attach should return properly as well.
I still think that maybe it is not an issue because the -i flag is intended if you want the container you are creating to accept input on it's stdin stream. As such the cli behaves properly by setting up a pipe for you to provide input to the container's stdin that you are attaching to. If the container wasn't intended to accept input on it's stdin then don't provide the -i flag to docker run
Thoughts?

As such the cli behaves properly by setting up a pipe for you to provide input to the container's stdin that you are attaching to.

I'm not saying the attach shouldn't be reading from STDIN. It should be. The container was launched with -i, and this is what it's supposed to do.

there are go routines in the daemon that are stuck waiting for input from the cli to forward on to the container you are attaching to. Hence why the cli is hanging and when you press a key, it will return to a normal state.

But the docker attach is also listens for output from the container. While it listens for output, it should also be watching for the container to exit. When it receives that exit event, it should stop waiting for input and exit, returning control to the parent.

If the container wasn't intended to accept input on it's stdin then don't provide the -i flag to docker run

In my example, sh -c 'sleep 2; echo exiting' is an arbitrary process. It could be replaced with anything, maybe something that can read STDIN, but it exited without doing so.

But just for arguments sake, lets run an interactive command. Try doing the following:
docker attach $(docker run -tid busybox sh)

Type a few things, run some commands, see that it works, then run exit. The process will exit, but you won't be returned to the shell you launched docker attach from (until you press another key).
Now what if you're attached to a container and you're waiting for it to do something, but instead it exits without a message. You're still going to be sitting at your docker attach waiting for it to do something, not knowing that it's not even running any more.

Also there is more severe bug about attach not detecting container events: #9797

@phemmer I have seen what you are talking about, sorry for my confusion. Think I found the culprit... working on a pull request now.

Closed by #10079.

can confirm this is fixed by #10079


Request: openflowj-loxi update in next major Floodlight release?

@rizard I know there is a major Floodlight release coming up shortly (v1.3?).
Not asking for any promises... but can you update to the latest openflowj-loxi when you push the release?  I'd like to be able to use the code from Loxigen PR 503 and Loxigen PR 505  if they are merged by the time you start the release process.
Thanks,
rsharo

@rsharo we're working on v2.0 given the many API changes. One big change will be removing ant support and using only maven. Maven will automatically fetch the latest OpenFlowJ-3.0.0-SNAPSHOT in v2.0 of Floodlight when you run mvn install. Master branch does this already.

@rizard Thanks!


fix(contrib/linode): fix issues with DHCP on Linode

Disable DHCP and mark the connection as critical on Linode. This has completely removed the network drops I was experiencing.

So UseDNS=false seems to mean that the resolver will ignore any nameserver entries returned by the DHCP response and stick with the hardcoded DNS= entries above it...correct? I had a hard time finding much info about this user-data stanza and its settings, so any further explanation would be helpful.
Regardless, I trust you've tested this and it helped your Linode deploys of Deis. Thanks a ton for keeping Deis contrib/ healthy @croemmich!

That's my understanding as well, but as you said, documentation is limited. For some reason the entries in the unit were being set up correctly on the initial run, but on the next lease they would get wiped out. UseDNS=false seems to have fixed the issue.
The most important part of this commit is CriticalConnection=true. I was having a very strange issue in which my ip was being dropped by the interface because the valid_lft on the address was not being updated even though DHCP leases were being issued successfully. See coreos/docs#377 for more info.
I've been running this configuration for about 90 days with success. (Which probably means it's working for all of the deployments on Linode :wink:)

I've been running this configuration for about 90 days with success.

Good enough for me. Thanks @croemmich!


Old People

Description
An "old person" (26 and 26+) should:
User Registration

[x] Should not be added to Mobile Commons
[ ] Not be added to the DoSomething Members list in MailChimp (mailchimp_list_id not equal to 71893 - ListID: f2fab1dfd4) and the DoSomething New Members grouping / UserRegistration interest group (ID# 10613)

[ ] User should be added to the Old People / 26+ Club MailChimp list (mailchimp_list_id equal to 80825 - ListID: a27895fe0c) and the "General" interest_grouping -> General Interest Group** (ID# 9613)

[ ] Send a confirmation transactional email to the old person confirming they have been subscribed to the 26+ Club list.

[ ] Send template name ?? in message payload. https://trello.com/c/rrrHgiZC

Campaign Signup

[x] Should not be added to Mobile Commons
[ ] Not be added to the DoSomething Members list in MailChimp (mailchimp_list_id not equal to 71893 ListID: f2fab1dfd4) and the Campaign Specific campaign grouping / interest group (values come from Drupal admin.

[ ] User should be added to the Old People / 26+ Club MailChimp list (mailchimp_list_id equal to 80825 - ListID: a27895fe0c) and the campaign specific campaign ?? -> ?? Interest Group (ID# ??) - managed by Drupal admin.

Technical Checklist
Mobile Commons

[x]  Move Mobile Commons Signup Submissions to Message Broker #3840
[x] Turn off Mobile Commons in MB payloads for persons older than 26 on registration and signups #4061

MailChimp

[ ] Make MailChimp List IDs manageable in Drupal on "Third Party Opt-Ins" page #3841
[ ] Extend MB payloads Drupal sends with MailChimp List IDs.
[ ] Replace MB MailChimp List ID setting in mbc-registration-email with the value from payload. Related issue: https://github.com/DoSomething/mbc-registration-email/issues/45
[ ] Send template name for Old People user registrations in message payload from Drupal (producer). Related: https://trello.com/c/rrrHgiZC
[ ] ...

Related:

https://github.com/DoSomething/mbc-registration-oldpeople/issues/1
https://trello.com/c/drdgqJt9

@barryclark @marahml How should we manage campaign MailChimp interest group assignment for Old People? Currently DS members are put into different MailChimp interest groups for each of the staff pick campaigns. Should there be duplicates in the 26+ list interest groups?

@sergii-tkachenko Updated issue description with the MailChimp list and interest group IDs for the 26+ list.

@DeeZone @sergii-tkachenko Dee and I chatted about this and confirmed that we should place all 26+ old people on to the Old People list only. They don't need to be added to a campaign specific group.
cc @marahml

@barryclark This means that we don't do anything on campaign signups? Just during registrations?

@sergii-tkachenko Correct.

Awesome. This make the whole deal a lot easier!

Just going to triple-check with @marahml in the HipChats too that she's cool with this.

As detailed in the issue description and repeated here for reference. The "General" interest_grouping (ID# 9613) should be used for MailChimp signups *but an "interest group" still need to be created with an appropreate name.
@barryclark @marahml - please create new "interest group" here vv.

https://us4.admin.mailchimp.com/lists/dashboard/groups?id=80825 (View Groups for "General").

@sergii-tkachenko, a proposed item for the checklist: After old people sign up, it'd be great if they could get a different confirmation message on site than the current one—so that we can let them know their signup is being handled differently. The current message is Registration successful. You are now logged in.. Is it possible to change that for old people? If so, I'll get a message put together and we can add it to the checklist. cc @DeeZone

FYI @sergii-tkachenko @barryclark -- We are currently using String Overrides to display that "Registration successful" message instead of Drupal's default message.

@barryclark What's the progress on the template for the message to be sent to 26+ user registrations?

we love old people now.


SD card as chroot target

Title says it all - feasible to have the chroot install onto an SD target?

I tried this on an Asus Chromebook Flip and the performance was horrible when installing a chroot to an SD card, despite it being a top-of-the-line Lexar 600x card. Depends on your hardware, I guess.


[FR] Popover preview for just the hovered highlight, and not all highlights.

Describe your request
When I hover over a particular highlight, it shows me a popover preview for all the highlights from the beginning of the markdown file, thus requiring me to scroll down all the way to find the hovered highlight. Thus, I wanted to request if there could be an option that shows just the selected highlight in the file.
I've just started using PDF++, so if there's already an option available that does this, I'd be happy to know about it. Thank you so much for making this excellent plugin, it has helped improve my workflow considerably and I appreciate all the effort behind it.

Hi, my apologies for being so late.
To be honest I'm a bit confused because PDF++ should automatically scroll down to the link's position in the markdown file.
For example:
https://github.com/user-attachments/assets/85fc9404-b703-4e5e-a0ca-db365118a0a7
If it doesn't work in your environment, it might be a bug. Can you consider posting a bug report if needed?


Drift web indexedDb slow

I am using the latest flutter and drift version in my project. When running on web (indexedDb) , it takes a lot of time to fetch the data from databse. IDB has onlt few entries in a table but each row has 6k of data in which one column consists of ~5k data in JSON text format,
This is how I have initialled the DB -
final db = Database(
WebDatabase.withStorage(
DriftWebStorage.indexedDb('worker',
migrateFromLocalStorage: false, inWebWorker: true),
),
);
db.customStatement('PRAGMA journal_mode=WAL');
Not sure why its so slow on web, on andorid and iOS it works fine

db.customStatement('PRAGMA journal_mode=WAL');

This isn't the reason, but WAL has no effect on the web.

IDB has onlt few entries in a table but each row has 6k of data in which one column consists of ~5k data in JSON text format,

Yes, drift is storing the database in 4k chunks. We're not reading that as JSON though.
What exactly is slow? Writes to the database or reads as well? Reads should be fairly fast because we keep the entire database in memory with IndexedDB. But writes in package:drift/web.dart are very slow and that is unfixable. You could also the new web implementation which uses a more sound implementation as well.

Thanks for the response, as I can see after adding the new way of implementation code, database transactions are faster now.
But on web, when I refresh the page, the database becomes empty. Any idea how to solve this? This was not happening before.
This is the code -
DatabaseConnection connectOnWeb() {
return DatabaseConnection.delayed(Future(() async {
final result = await WasmDatabase.open(
databaseName:
'myDb', // prefer to only use valid identifiers here
sqlite3Uri: Uri.parse('sqlite3.wasm'),
driftWorkerUri: Uri.parse('drift_worker.dart.js'),
);
if (result.missingFeatures.isNotEmpty) {
  debugPrint('Using ${result.chosenImplementation} due to missing browser '
      'features: ${result.missingFeatures}');
}

return result.resolvedExecutor;

}));
}

Interesting, I haven't seen that before. Do you see a message about it using an in-memory implementation due to missing browser features? Or, which implementation is mentioned in the debugPrint message?

So its not going in the IF condition. I am using latest chrome and if I remove the if condition and try to run debugPrint directly - this is the output it shows
Using WasmStorageImplementation.inMemory due to missing browser features: {}

That sounds like something is going very wrong, is there any error before that? Do you have the drift_worker.dart.js file in web/? That message can appear when drift was unable to load the worker.


feat(intersect) added intersect function …

it creates a unique array of values that is contained in both arrays.

Cheers let me know if there's anything else i can do :)

@dev-warner Thank you for the refinements Joe! Looks good, merged!


docs(operators): Added docs/examples for isEmpty

Description:
Added documentation for isEmpty operator

@martinsik all of your comments have been resolved, thanks for reviewing 😃

Pull Request Test Coverage Report for Build 8133

1 of 1 (100.0%)  changed or added relevant line in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 96.805%

Totals

Change from base Build 8098:
0.0%

Covered Lines:
5242

Relevant Lines:
5415

💛  - Coveralls

Fixed the commit message warning.


Media Fragments URI

Media Fragments URI http://www.w3.org/TR/media-frags

This issue was imported from Google Moderator
Moderator votes: +16
Issue added by Josh T. on 2011-09-03
To vote this issue up or down, simply include +1 or -1 in your comment.

+1

+1

+1

+1

Now available at http://caniuse.com/#feat=media-fragments