Add mapping slot API to cheatcodes

Motivation
When testing a contract, we have no knowledge about how many elements there're in a mapping, so we'd like to add a family of cheatcode to solve this problem.
Solution

we add these api to Cheatcodes

            startMappingRecording()
            getMappingLength(address,bytes32)
            getMappingSlotAt(address,bytes32,uint256)
            getMappingKeyOf(address,bytes32)
            getMappingParentOf(address,bytes32)

after startMappingRecording() we record all potential map sstore to Cheatcodes local state
getMappingLength(address target, bytes32 mappingSlot) returns (uint256) would get count of elements in mappingSlot at target address.
getMappingSlotAt(address target, bytes32 mappingSlot, uint256 idx) returns (bytes) would get elements of idx in the mapping 0 <= idx < length, the length is returned by getMappingLength
getMappingKeyOf(address target, bytes32 elementSlot) returns (bytes) and getMappingParentOf(address target, bytes32 elementSlot) returns (bytes) are called on "element slot" returned by getMappingSlotAt

Details

we would record on sha3 and sstore on step
when see a slot in sstore, we would first check if it is generated by some sha3. If so, this would probable be a element of some mapping.

When testing a contract, we have no knowledge about how many elements there're in a mapping,
Seems like this would be a useful cheatcode

wdyt @mds1 ?

https://github.com/foundry-rs/foundry/issues/4635

This is a really interesting idea, thanks @clouds56! Being able to loop over mapping keys is very useful, especially during invariant tests, so people often use arrays to manually track all keys written to a mapping to work around this limitation.
I want to think on this API a bit before merging, as I'm not yet convinced it's the best UX. Some scattered initial thoughts for now:

Having to use slot numbers isn't ideal since they can change as contract development progresses
Having to use slot numbers isn't ideal since it's not very user friendly, it would be nice if you can reference mappings by their variable name, similar to the StdStorage UX in forge-std.
Using variable names would likely require knowing the storage layout (and having forge compile with that enabled by default). But maybe that's ok / a better route, because it would also unlock the ability to add cheats to make it easier to write to packed slots (ref https://github.com/foundry-rs/foundry/issues/3869#issuecomment-1363172637).

Having to use slot numbers isn't ideal since it's not very user friendly, it would be nice if you can reference mappings by their variable name, similar to the StdStorage UX in forge-std.

I agree with this is not the best UX, but I think it is also important that we could have simple and powerful API in vm.
Since not all mappings has a "name", think of a nest mapping
mapping(uint => mapping(uint => uint)) d;

how to name d[1] which is also a map?
Shall we let StdStorage handle the name resolving things (maybe a separate PR) while let vm simple?
A possible StdStorage API could be like:
// d[1][2] = 3
// `read_mapping` would set storage dynamic type to mapping
StdStorage root = stdstore.target(address(test)).sig("d()").read_mapping();
// `getLength` would call `getMappingLength` on correspond slot
for (uint i = 0; i < root.getLength(); i++) {
  // `element` would call `getMappingSlotAt`
  StdStorage sub = root.element(i).read_mapping();
  for (uint j = 0; j < sub.getLength(); j++) {
    uint value = root.element(j).read_u256();
  }
}

inner StdStorage, these vm API are actually called
            getMappingLength(address,bytes32)
            getMappingSlotAt(address,bytes32,uint256)

I implemented root() and parent() in StdStorage.
Now we could use
(uint256 slot, bytes32 key) = stdstore.target(address(test)).sig(test.map_addr.selector).with_key(address(this)).parent();
// or
stdstore.target(address(test)).sig(test.map_addr.selector).with_key(address(this)).root();

to get root slot.
Note: you have to pass right number of .with_key, while the key doesn't have to exists. See https://github.com/foundry-rs/forge-std/pull/343 for more details

Sorry for the slow response here, busy week. Lots of questions below.

mapping(uint => mapping(uint => uint)) d;

how to name d[1] which is also a map?

You could do it just like that, that seems like a good convention, i.e. d[0] would be the first mapping key, d[1] the second key, and if there's only a single key then d and d[0] would equivalent.

Shall we let StdStorage handle the name resolving things (maybe a separate PR) while let vm simple?

I do think we should make as much of this native to forge as we can, per https://github.com/foundry-rs/foundry/issues/3782, so if we are able to avoid needing a forge-std PR that would be preferable. Rust is a lot more flexible than solidity too, so handling things here seems simpler than having to handle them in solidity. Is there a reason you prefer to split things up into forge-std?
Also can you provide some examples of what it would look like to use the full set of cheats + forge-std to get an array of all mapping keys for a regular mapping and nested mapping? Want to make sure I fully understand the UX here, there's a lot of cheats and I'm not sure I fully understand all the nomenclature yet, e.g. parent, root, etc.
One thing in particular is, instead of a cheat to return the length, should we additionally have a family of cheats like the below? It feels like that would be better UX since I imagine getting the full array is the most common use case, and of course you can always easily get the length from there.

getMappingKeysUint(...) returns (uint[] memory)
getMappingKeysAddress(...) returns (address[] memory)
getMappingKeysBytes(...) returns (bytes[] memory)  -- for custom types that aren't bool/uint/int/bytes32/bytes/string/address

It's probably keep to keep the length + index versions for better performance in cases where the arrays get really big.
Should there be a stopMappingRecording() cheat also? Wondering if recording all this can get costly / come with a performance penalty, so you might want the ability to turn it on/off.
Lastly, how do the getters behave in terms of their buffers? For example, recordLogs starts recording logs, then getRecordedLogs returns all recorded logs then clears it's buffer. So an immediate subsequent call to getRecordedLogs returns nothing. We have 4 getters here, so I assume we can't immediately clear the buffer as soon as one is called, and that the lengths only grow?

I'm not familiar with UX things, the new adding
getMappingKeysUint(...) returns (uint[] memory)
getMappingKeysAddress(...) returns (address[] memory)
getMappingKeysBytes(...) returns (bytes[] memory)

could all built on top of current APIs.

I do think we should make as much of this native to forge as we can, per https://github.com/foundry-rs/foundry/issues/3782, so if we are able to avoid needing a forge-std PR that would be preferable.
Is there a reason you prefer to split things up into forge-std?

I like the idea to get rid of forge-std, while if i read it correctly, the #3782 is just a draft RFC, it even hasn't reached the final design. We could not let existing PR to follow a standard not exists.
The reason I prefer split is now things are handled in this way, I'm just making consistence to existing code base.

Should there be a stopMappingRecording() cheat also?

I agree with we should add stopMappingRecording, given we have record(), recordLogs(), I think you should add stopRecord() and stopRecordLogs() as well.
I think the UX of contribution is some how not the best for now? I think we could:

let the basic API merged if there's no bug (since any other API could be built upon current API, do you agree with this @mattsse?)
gives a final design of implementable API (no matter forge-std or vm itself), and maybe I could do that in a separate PR if I have time.
migrate current code to "the better UX" you mean ASAP, so following contributor could learn the best practice by reading the code.

As a side note for stdstore:
Why couldn't we get slot from layout, forge inspect is layout aware, why can't stdstore get this information?
$ forge inspect src/Greeter.sol:Greeter storage --pretty
| Name     | Type                                           | Slot | Offset | Bytes | Contract                |
|----------|------------------------------------------------|------|--------|-------|-------------------------|
| greeting | string                                         | 0    | 0      | 32    | src/Greeter.sol:Greeter |
| owner    | address                                        | 1    | 0      | 20    | src/Greeter.sol:Greeter |
| data     | mapping(address => uint256)                    | 2    | 0      | 32    | src/Greeter.sol:Greeter |
| data2    | mapping(address => mapping(address => int256)) | 3    | 0      | 32    | src/Greeter.sol:Greeter |

According to my experience here, I think the interface design is quite ergonomic.
Do you have any comments and suggestions for subsequent updates and modifications?

I agree with we should add stopMappingRecording, given we have record(), recordLogs(), I think you should add stopRecord() and stopRecordLogs() as well.

this sounds reasonable to me

let the basic API merged if there's no bug (since any other API could be built upon current API, do you agree with this

defer to @mds1 here

let the basic API merged if there's no bug (since any other API could be built upon current API, do you agree with this @mattsse?)

Ok, I'm good with that. I believe you're right we should be able to build the UX improvements on top of this base, and this is better than the existing solution of having to manually implement/track everything anyway

As a side note for stdstore: Why couldn't we get slot from layout, forge inspect is layout aware, why can't stdstore get this information?

By default solc/forge does not include the storage layout in the build artifacts. So we'd either have to:

Always enable storage layout as an output by default. This might increase compilation times (or maybe not, and it's roughly free), so we'd have to check that first
Revert if the artifact doesn't have storage layout and explain it needs to be enabled

But otherwise you're right that's were we'd get the info from. forge inspect just recompiles with the storage layout output enabled to get the data

According to my experience here, I think the interface design is quite ergonomic.

This is a good example, thank you.

Do you have any comments and suggestions for subsequent updates and modifications

Here's my suggested plan. First for this PR:

Merge it (pending any comments @mattsse might have on code)
Open a PR to the book repo for docs
Open a PR to forge-std to add the cheats to VmSafe in Vm.sol

Then create a separate issue to tract UX improvements using mapping names, where we should:

Take a few repos that use forge (should do seaport with via-ir, forge-std, maple labs, uniswaps universal router and permit2 are all good options). Use hyperfine to benchmark the difference in compilation with and without storage layout as an output. It's likely solc already has the data so it's cheap to spit out, or it's possible it involves some extra work with nontrivial added time to compilation
If it's cheap, let's include it by default in forge for better UX. Having access to storage layout will help a lot with these cheats + writing to packed slots. If it's not cheap, the rest of the steps are the same, but if the new UX-friendlier cheats are used without storage layout they just revert and explain why
Add cheats that let you specify vars/slots by name instead of index

Thanks a lot for this PR! This will be a great improvement, and I appreciate your patience around the UX questions/decisions :)

i've just stumbled upon this.. absolutely banger feature

Agreed! @clouds56 any chance you could resolve the conflicts here and maybe we can get this merged this week?

@clouds56 I'm unable to push directly to this branch (perhaps you don't allow changes by maintainers?)
I pushed to https://github.com/foundry-rs/foundry/pull/5123 feel free to update this PR with that and we can close https://github.com/foundry-rs/foundry/pull/5123

merged thru #5123


help: Create channels and user groups instructions for UI updates.

Notes:

Expanded "info" to "information".
Notes about where the "Create" button was seemed redundant since the user will have clicked the "Continue to add ..." buttons that are in the same place and have very clear/particular text.

Fixes #30861.
Screenshots and screen captures:

Create channels instructions
Current documentation

Create user groups instructions
Current documentation

Self-review checklist

[ ] Self-reviewed the changes for clarity and maintainability
(variable names, code reuse, readability, etc.).

Communicate decisions, questions, and potential concerns.

[ ] Explains differences from previous plans (e.g., issue description).
[ ] Highlights technical choices and bugs encountered.
[ ] Calls out remaining decisions and concerns.
[ ] Automated tests verify logic where appropriate.

Individual commits are ready for review (see commit discipline).

[ ] Each commit is a coherent idea.
[ ] Commit message(s) explain reasoning and motivation for changes.

Completed manual review and testing of the following:

[ ] Visual appearance of the changes.
[ ] Responsiveness and internationalization.
[ ] Strings and tooltips.
[ ] End-to-end functionality of buttons, interactions and flows.
[ ] Corner cases, error conditions, and easily imagined bugs.

Works for me!

Merged, thanks @laurynmm!


Improved development setup

Several changes to improve our dev setup: removed a bunch of unused dependencies, fixed circular imports, removed a few functions that I don't think we use, and ensure via the doc that we use the same node and parcel versions.
To test this branch, please:

Pull it
Remove dependencies: delete your node_modules folder
Install and use the node version recommended in the new README: node LTS, i.e. v18.13.0 at the time of this writing
Re-install the dependencies: run npm install
Use the npx parcel@latest command recommended in the new README

LGTM


"German" reported as Country, vs. nationality.

version Xponents 3.4 beta.

German is correctly tagged as nationality.
German is omitted from gazetteer as a A/PCLI feature.  Germany and other variants are fine.


DotNetAspire: This service descriptor is keyed. Your service provider may not support keyed services

Hey, when using MediatR with Aspire I seem to get this issue (it works perfectly fine without it though).
Unhandled exception. System.InvalidOperationException: This service descriptor is keyed. Your service provider may not support keyed services.
   at Microsoft.Extensions.DependencyInjection.ServiceDescriptor.ThrowKeyedDescriptor()
   at Microsoft.Extensions.DependencyInjection.ServiceDescriptor.get_ImplementationType()
   at MediatR.Registration.ServiceRegistrar.<>c.<RegisterBehaviorIfImplementationsExist>b__12_0(ServiceDescriptor service)
   at System.Linq.Enumerable.SelectIListIterator`2.MoveNext()
   at System.Linq.Enumerable.WhereEnumerableIterator`1.MoveNext()
   at System.Linq.Enumerable.SelectManySingleSelectorIterator`2.MoveNext()
   at System.Linq.Enumerable.WhereSelectEnumerableIterator`2.MoveNext()
   at System.Linq.Enumerable.WhereEnumerableIterator`1.MoveNext()
   at System.Linq.Enumerable.Any[TSource](IEnumerable`1 source, Func`2 predicate)
   at MediatR.Registration.ServiceRegistrar.RegisterBehaviorIfImplementationsExist(IServiceCollection services, Type behaviorType, Type subBehaviorType)
   at MediatR.Registration.ServiceRegistrar.AddRequiredServices(IServiceCollection services, MediatRServiceConfiguration serviceConfiguration)
   at Microsoft.Extensions.DependencyInjection.ServiceCollectionExtensions.AddMediatR(IServiceCollection services, Action`1 configuration)
   at OctoMenu.Backend.Menu.Application.ConfigureServices.AddBackendServices(IServiceCollection services) in ...\src\libs\backend\Menu.Application\ConfigureServices.cs:line 17
   at Program.<Main>$(String[] args) in ...\src\apps\menu-api\Program.cs:line 18

I'm registering like so:

  public static IServiceCollection AddBackendServices(this IServiceCollection services)
  {
    var assembly = Assembly.GetExecutingAssembly();
    services.AddValidatorsFromAssembly(assembly);
    services.AddMediatR(cfg =>
    {
      cfg.RegisterServicesFromAssembly(assembly);
      cfg.AddBehavior(typeof(IPipelineBehavior<,>), typeof(ValidationBehaviour<,>));
    });
    services.AddTransient<ProductMenuDbContext, ProductMenuDbContext>();
    return services;
  }
}

This was working just fine before I started using .NET Aspire do you know if there are any compatibility issues?

Okay literally just after writing this I discovered it's an issue with .NET Aspire specifically.
I solved it by downgrading a package:  Microsoft.Extensions.Http.Resilience to version 8.0.0 and it worked.


Update to spring-cloud-deployer 1.0.0.RC1

this was already merged via https://github.com/spring-cloud/spring-cloud-deployer-mesos/commit/0531b2a4152b0b71538890a74908134cd97da553


add GCC option for checking virtual table pointers

add GCC option for checking virtual table pointers

New pull request to add a section about virtual table checking section after synchronizing my repository with the base branch.

@thomasnyman @david-a-wheeler This pull request is a duplication of #440. I close my first pull request by mistake during the synchronization of my repository.
Sorry about this mistake.
Regards,

@Flo4152 I've opened a pull request against your branch to address the linter errors as well as some consistency issues with  the rest of the guide: https://github.com/Flo4152/wg-best-practices-os-developers/pull/1
Given the observation by @thesamesam in his https://github.com/ossf/wg-best-practices-os-developers/pull/485#discussion_r1594479772 regarding ABI breakage I think that is something that really warrants as When to not use? section I was asking for in the original PR
I am also wondering whether -fsanitize=vptr combined with the -fsanitize-minimal-runtime we were pointed to look at in https://github.com/ossf/wg-best-practices-os-developers/issues/326 would be a better options for production code.
That said, I do think the material on the usage of -fvtable-verify in this PR is good and given that there are examples of users of this, even though it may not be fit for a general recommendation, I do think this would be good material for the guide as long as the caveats with the option are clearly communicated.

My reservation is that I've genuinely never heard of anyone using this outside of those initial talks when it got merged into GCC. I'm sure users have existed, or do exist somewhere, but I've never heard from them and I don't ever see it mentioned in GCC bug reports or development discussions.
I'd feel a lot better about advertising this if:

someone could ask around the GCC community (perhaps on the ML) what the consensus on the option is, and if it's suitable for the guide;
someone could try build a bunch of software with it and see what happens (and make sure e.g. the compiler doesn't just ICE immediately)

These are really significant performance impacts, on a relatively narrowly focused countermeasure.
We definitely should document these options, including the performance impacts noted above. I don't think we should recommend using these options by default.

@david-a-wheeler I describe performance impacts in my last change of "Performance implications" subsection. I think I summarized number from SPECS benchmarks. Do you think there's anything missing or should be completed?
I agree theses options should not recommend using these options by default. Do you mean these options should be moved from the "Recommended compiler options that enable run-time protection mechanisms." table?

Proposal - include text, but include at the end something like: “This option can have significant performance impact, while only countering a narrow type of vulnerability. Thus, we have not included this option as a recommended option in the current version of this guide.”

I'll stop after this and won't insist further, but I have to ask:

We definitely should document these options, including the performance impacts noted above. I don't think we should recommend using these options by default.

Why is this a definitely? Including things which we're saying people shouldn't use by default (and with a huge amount of caveats) is noise and it may end up reducing the effectiveness of the guide if it's a huge document with lots of irrelevant parts.
There are other options which have significant performance impact which are worth mentioning because they're actively developed and tested at least. But to me, this particular one appears all-negative (https://github.com/ossf/wg-best-practices-os-developers/pull/485#issuecomment-2118983435). There are various options which are also not-great ideas which I could make PRs for. What makes this one different?
Isn't part of the goal here to focus on modern, useful, and sensible options?

@thomasnyman @david-a-wheeler There are arguments against adding this flag to your guide. This pull request was discussed at your zoom calls, and adding this flag seems to be in line with what was said at those calls.
Do you still feel the same way about adding this flag?

It appears that the current state is the following:

The -fvtable-verify option was implemented long ago with the idea that it may be useful, but never got any use in the Open Source ecosystem. Further, it appears that the feature is largely unmaintained since the feature implementors have since stopped contributing to GCC.
The option in isolation breaks ABI, so it's unlikely that an individual project would be able to use it safely in any capacity.
The option has a significant performance overhead

All of that sounds to me like this shouldn't go into the main guide, but should get mentioned in the Appendix as an option that was considered but rejected.

In the interest of trying to wrap this topic up after a lengthy discussion, I've proposed some text to add to the list of considred options for -fvtable-verifyin #553.

@thomasnyman @siddhesh I agree. I'm closing this request.

Thank you all for your responses to this pull request. I appreciate the corrections, advice and help I received.
Regards,


Lists not supported in properties file

It is not possible to enter multiple values for swagger2markup.pageBreakLocations.  From the code, it looks like enabling lists is done after all the properties are read in.

My mistake. I meant to create this issue in swagger2markup project.


Timestamps once again

I have just started using objection.js and I love it!
However, I have a problem with timestamps. It is not about how to create them. I followed the docs and looked up examples and they are setup properly.
The problem is that I want those fields to be set only by the db, and from none else.
Instead, after the setup I did, I do
People.query().insert({updated_at: '2017-01-01'});

I am able to explicitly set the value. Is there a way to avoid this?
My people.js model file:
'use strict';

const Model = require('objection').Model;

class People extends Model {
  // Table name is the only required property.
  static get tableName() {
    return 'people';
  }

  $beforeInsert() {
    this.created_at = new Date().toISOString();
  }

  $beforeUpdate() {
    this.updated_at = new Date().toISOString();
  }
  static get jsonSchema() {
    return {
      type: 'object',
      required: ['first_name', 'last_name'],

      properties: {
        id: { type: 'integer' },
        first_name: { type: 'string', minLength: 1, maxLength: 255 },
        last_name: { type: 'string', minLength: 1, maxLength: 255 },
        bio: { type: 'string' }
      }
    }
  };
}
module.exports = People;

and my migration file for the people table:
'use strict'

exports.up = function (knex, Promise) {
  return knex.schema
    .createTable('people', function (table) {
      table.increments('id').primary();
      table.string('first_name').notNullable();
      table.string('last_name').notNullable();
      table.text('bio');
      table.timestamps(true, true);
    });
};

exports.down = function (knex, Promise) {
  return knex.schema.dropTable('people');
};

You can do this:
  $beforeInsert() {
    this.created_at = new Date().toISOString();
    delete this.updated_at;
  }

  $beforeUpdate() {
    this.updated_at = new Date().toISOString();
    delete this.created_at;
  }

or is there something that prevents you from doing that?

No, I didn't think about that. I'll give it a try and if it works fine i'll close the issue.
Thanks!

@koskimas ok, so, this works but I think it's kind of a "trick" because it relies on the webserver to set the datetime, when I think that it should be the db itself. Don't you think?

That's easily doable using DEFAULT keyword in the migration for created_at . Something like this:
table.timestamp('created_at').defaultTo(knex.raw('now()'));

You need to check the correct syntax for now() for your database engine. Autmatic updated_at can be achieved using triggers. I don't remember the correct syntax off the top of my head, but google is your friend. Then if you want to make absolutely sure the code never sets those columns you can override $formatDatabaseJson:
$formatDatabaseJson(json) {
  json = super.$formatDatabaseJson(json);
  delete json.created_at;
  delete json.updated_at;
  return json;
}

In any case, you need to do this in the database and you cannot do it using objection. Objection has no magic in it that automatically generates anything in the database. It simply uses the schema, but never modifies it.

Something like this would work on postgres:
CREATE OR REPLACE FUNCTION update_updated_at_column()
RETURNS TRIGGER AS $$
BEGIN
    NEW.updated_at = now();
    RETURN NEW;
END;
$$ language 'plpgsql';

CREATE TRIGGER update_people_updated_at BEFORE UPDATE ON people FOR EACH ROW EXECUTE PROCEDURE  update_updated_at_column();

You need to use knex.raw to create those functions and triggers in a migration file.

@koskimas Thank you very much.
I have just a question given that you initially proposed a solution which dealt with timestamps in the webapp itself.
From a design point of view, do you think that dealing with created_at and updated_at in the webapp instead of the db is correct?

I think its better to do it in the database since you can be absolutely sure the value is always updated, but doing it using triggers is a bit clumsy since you need to write raw sql and the code is different for each database. That's why the recipe is to do it in the webapp. There's nothing wrong in that either, and it works just fine. It only fails if you go around objection models and use raw SQL or knex directly.

ok. thank you very much, i'm gonna close the issue

It should work

hmm.. ok here is my full chain:

./knexfile.js

const { knexSnakeCaseMappers } = require('objection')
require('dotenv').config({ path: './.env' })

console.log('knex connecting to', process.env.PSQL_HOST) // eslint-disable-line

module.exports = Object.assign(
  {
    client: 'pg',
    connection: {
      host: process.env.PSQL_HOST,
      database: process.env.PSQL_DATABASE,
      user: process.env.PSQL_USERNAME,
      password: process.env.PSQL_PASSWORD,
    },
    migrations: {
      stub: './knex-stub.js',
    },
  },
  knexSnakeCaseMappers()
)

./db/models/BaseModel.js

const Knex = require('knex')
const objection = require('objection')
const config = require('../../knexfile')

/**
 * Extend objection Model to include a property for
 * defining a PostgreSQL schema and using it in queries.
 */
class ExtendedQueryBuilder extends objection.QueryBuilder {
  constructor(ModelClass) {
    super(ModelClass)
    if (ModelClass.schemaName) {
      this.withSchema(ModelClass.schemaName)
    }
  }
}

objection.Model.QueryBuilder = ExtendedQueryBuilder
objection.Model.RelatedQueryBuilder = ExtendedQueryBuilder

objection.Model.knex(Knex(config))

module.exports = class BaseModel extends objection.Model {
  static get modelPaths() {
    return [__dirname]
  }

  $beforeInsert() {
    if (this.timestamps) {
      const timestamp = new Date().toISOString()
      this.createdAt = timestamp
      this.updatedAt = timestamp
    }
  }

  $beforeUpdate() {
    if (this.timestamps) {
      this.updatedAt = new Date().toISOString()
    }
  }
}

./db/models/User.js

const BaseModel = require('./BaseModel')

module.exports = class User extends BaseModel {
  static get schemaName() {
    return 'auth'
  }
  static get tableName() {
    return 'users'
  }
  static get timestamps() {
    return true
  }

  static get relationMappings() {
    return {
      emails: {
        relation: BaseModel.HasManyRelation,
        modelClass: 'Email',
        join: {
          from: 'users.id',
          to: 'emails.userId',
        },
      },
      groups: {
        relation: BaseModel.ManyToManyRelation,
        modelClass: 'Group',
        join: {
          from: 'users.id',
          through: {
            from: 'users_groups.userId',
            to: 'users_groups.groupId',
            extra: ['role'],
          },
          to: 'groups.id',
        },
      },
    }
  }
}

./db/models/Email.js

const BaseModel = require('./BaseModel')

module.exports = class Email extends BaseModel {
  static get schemaName() {
    return 'auth'
  }
  static get tableName() {
    return 'emails'
  }
  static get timestamps() {
    return true
  }

  static get relationMappings() {
    return {
      user: {
        relation: BaseModel.BelongsToOneRelation,
        modelClass: 'User',
        join: {
          from: 'emails.userId',
          to: 'users.id',
        },
      },
    }
  }
}

./db/models/Group.js

const BaseModel = require('./BaseModel')

module.exports = class Group extends BaseModel {
  static get schemaName() {
    return 'auth'
  }
  static get tableName() {
    return 'groups'
  }
  static get timestamps() {
    return true
  }

  static get relationMappings() {
    return {
      users: {
        relation: BaseModel.ManyToManyRelation,
        modelClass: 'User',
        join: {
          from: 'groups.id',
          through: {
            from: 'users_groups.groupId',
            to: 'users_groups.userId',
            extra: ['role'],
          },
          to: 'users.id',
        },
      },
    }
  }
}

Running a query:
// Graph insert:
(async () => {
  console.log(
    await User.query().insertGraph({
      emails: [{
        email<EMAIL_ADDRESS>        verified: true
      }],
      groups: [{
        name: 'Ethos',
        type: 'other',
        role: 'admin'
      }],
   })
  )
})()

Result:
(node:14482) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): error: insert into "auth"."users" default values returning "id" - null value in column "created_at" violates not-null constraint
(node:14482) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

Nevermind, I got it to work, I have no idea why it didn't, probably because I was using the node REPL and didn't drop the require cache before trying the query again.
Thanks!!!

Ah, it didn't work because in my BaseModel:
$beforeInsert() {
    if (this.timestamps) { // <-- Should actually be `this.constructor.timestamps`
      const timestamp = new Date().toISOString()
      this.createdAt = timestamp
      this.updatedAt = timestamp
    }
  }

  $beforeUpdate() {
    if (this.timestamps) { // <-- Should actually be `this.constructor.timestamps`
      this.updatedAt = new Date().toISOString()
    }
  }

... in case anyone else finds their way to this issue.

how about
class YourModel extends ObjectionModel {
  static get jsonSchema() {
    return {
      type: `object`,
      additionalProperties: false, // will throw error if there is additional properties
      properties: {
        ...your properties schema,
      }
    }
  }
}

and if you want to not throw error but remove the additional properties instead, use this
static createValidator() {
    return new AjvValidator({
      onCreateAjv: ajv => {}, // if want to custom something
      options: {
        useDefaults: true,
        removeAdditional: true, // will remove additional properties if you set schema additionalProperties: fase
        allErrors: true,
        validateSchema: false,
        ownProperties: true,
        v5: true,
      },
    });
  }

https://vincit.github.io/objection.js/#ajvvalidator
https://github.com/epoberezkin/ajv#filtering-data

I like the example by @heisian. There is also https://github.com/oscaroox/objection-timestamps.
Rather than using new Date().toISOString(), I use this.$knex().fn.now().


Memory spike when using broker with batching policy

Hi, I'm trying to enable a batching policy that will allow me to generate 40MB CSVs from a large json lines (jsonl) file. While testing this I noticed that Benthos was getting OOM-killed consistently by docker as it exceed 2GB in memory usage. I managed to strip away my config to something that reproduces the issue and have captured it below. If I remove the broker that is around the drop output the pipeline runs successfully to completion. The issue also occurs I use a batching broker at the input instead of output.

Command
docker run --rm --name readdrop -v $PWD/data:/data -v $PWD/project:/project jeffail/benthos:3.60.1 -c /project/config.yml

Config
input:
  file:
    paths: [/data/input/pypi/*.jsonl]
    codec: lines
  processors:
    - bloblang: |
        map escape_csv {
          root = if this.re_match("[\"\n,]+") {
            "\"" + this.replace("\"", "\"\"") + "\""
          } else {
            this
          }
        }

        meta = deleted()

        root = [
          (this.timestamp | "").string().apply("escape_csv"),
          (this.project | "").string().apply("escape_csv"),
          (this.url | "").string().apply("escape_csv"),
          (this.details.installer.name | "").string().apply("escape_csv"),
          (this.details.installer.version | "").string().apply("escape_csv"),
          (this.details.implementation.name | "").string().apply("escape_csv"),
          (this.details.implementation.version | "").string().apply("escape_csv"),
          (this.details.system.name | "").string().apply("escape_csv"),
          (this.details.system.release | "").string().apply("escape_csv")
        ].join(",")
output:
  broker:
    outputs:
      - drop: {}
    batching:
      period: 30s
      byte_size: 40_000_000

Sample data
There is only one 942.2 MB file that is picked up by the file input and it is formatted as JSON lines. It's public data about PyPI requests that was available as a table in one of the public BigQuery datasets.
{"timestamp":"2021-11-29 00:41:50 UTC","country_code":"US","url":"/packages/e5/18/401594af67eda194a8b9167208621761927c937db7d60292608342bbac0a/pytest_cov-2.10.1-py2.py3-none-any.whl","project":"pytest-cov","file":{"filename":"pytest_cov-2.10.1-py2.py3-none-any.whl","project":"pytest-cov","version":"2.10.1","type":"bdist_wheel"},"details":{"installer":{"name":"pip","version":"21.3.1"},"python":"3.9.9","implementation":{"name":"CPython","version":"3.9.9"},"distro":{"name":"Ubuntu","version":"20.04","id":"focal","libc":{"lib":"glibc","version":"2.31"}},"system":{"name":"Linux","release":"5.11.0-1021-azure"},"cpu":"x86_64","openssl_version":"OpenSSL 1.1.1f  31 Mar 2020","setuptools_version":"59.3.0"},"tls_protocol":"TLSv1.2","tls_cipher":"ECDHE-RSA-AES128-GCM-SHA256"}
{"timestamp":"2021-11-29 15:02:12 UTC","country_code":"US","url":"/packages/3f/12/d5089675aa7955c9b7e8b831184b2b13ccece146c5b23250b0102a0d76e2/pytest_cov-2.11.0-py2.py3-none-any.whl","project":"pytest-cov","file":{"filename":"pytest_cov-2.11.0-py2.py3-none-any.whl","project":"pytest-cov","version":"2.11.0","type":"bdist_wheel"},"details":{"installer":{"name":"pip","version":"21.3.1"},"python":"3.6.8","implementation":{"name":"CPython","version":"3.6.8"},"distro":{"name":"Ubuntu","version":"20.04","id":"focal","libc":{"lib":"glibc","version":"2.31"}},"system":{"name":"Linux","release":"5.11.0-1021-azure"},"cpu":"x86_64","openssl_version":"OpenSSL 1.1.1f  31 Mar 2020","setuptools_version":"40.6.2"},"tls_protocol":"TLSv1.2","tls_cipher":"ECDHE-RSA-AES128-GCM-SHA256"}
{"timestamp":"2021-11-29 07:17:05 UTC","country_code":"US","url":"/packages/3f/12/d5089675aa7955c9b7e8b831184b2b13ccece146c5b23250b0102a0d76e2/pytest_cov-2.11.0-py2.py3-none-any.whl","project":"pytest-cov","file":{"filename":"pytest_cov-2.11.0-py2.py3-none-any.whl","project":"pytest-cov","version":"2.11.0","type":"bdist_wheel"},"details":{"installer":{"name":"pip","version":"20.3.3"},"python":"3.8.12","implementation":{"name":"CPython","version":"3.8.12"},"distro":{"name":"Ubuntu","version":"20.04","id":"focal","libc":{"lib":"glibc","version":"2.31"}},"system":{"name":"Linux","release":"5.11.0-1021-azure"},"cpu":"x86_64","openssl_version":"OpenSSL 1.1.1f  31 Mar 2020","setuptools_version":"58.3.0"},"tls_protocol":"TLSv1.2","tls_cipher":"ECDHE-RSA-AES128-GCM-SHA256"}
{"timestamp":"2021-11-29 18:41:37 UTC","country_code":"US","url":"/packages/3f/12/d5089675aa7955c9b7e8b831184b2b13ccece146c5b23250b0102a0d76e2/pytest_cov-2.11.0-py2.py3-none-any.whl","project":"pytest-cov","file":{"filename":"pytest_cov-2.11.0-py2.py3-none-any.whl","project":"pytest-cov","version":"2.11.0","type":"bdist_wheel"},"details":{"installer":{"name":"pip","version":"21.0.1"},"python":"3.8.10","implementation":{"name":"CPython","version":"3.8.10"},"distro":{"name":"Ubuntu","version":"20.04","id":"focal","libc":{"lib":"glibc","version":"2.31"}},"system":{"name":"Linux","release":"5.11.0-1021-azure"},"cpu":"x86_64","openssl_version":"OpenSSL 1.1.1f  31 Mar 2020","setuptools_version":"44.0.0"},"tls_protocol":"TLSv1.2","tls_cipher":"ECDHE-RSA-AES128-GCM-SHA256"}
{"timestamp":"2021-11-29 06:07:02 UTC","country_code":"US","url":"/packages/e3/1a/6affecd2344efee7f2487fac82242474cbac09f9e04929da5944907baf11/pytest_cov-2.11.1-py2.py3-none-any.whl","project":"pytest-cov","file":{"filename":"pytest_cov-2.11.1-py2.py3-none-any.whl","project":"pytest-cov","version":"2.11.1","type":"bdist_wheel"},"details":{"installer":{"name":"pip","version":"21.3.1"},"python":"3.7.12","implementation":{"name":"CPython","version":"3.7.12"},"distro":{"name":"Ubuntu","version":"20.04","id":"focal","libc":{"lib":"glibc","version":"2.31"}},"system":{"name":"Linux","release":"5.11.0-1021-azure"},"cpu":"x86_64","openssl_version":"OpenSSL 1.1.1f  31 Mar 2020","setuptools_version":"47.1.0"},"tls_protocol":"TLSv1.2","tls_cipher":"ECDHE-RSA-AES128-GCM-SHA256"}
{"timestamp":"2021-11-29 13:10:42 UTC","country_code":"US","url":"/packages/e3/1a/6affecd2344efee7f2487fac82242474cbac09f9e04929da5944907baf11/pytest_cov-2.11.1-py2.py3-none-any.whl","project":"pytest-cov","file":{"filename":"pytest_cov-2.11.1-py2.py3-none-any.whl","project":"pytest-cov","version":"2.11.1","type":"bdist_wheel"},"details":{"installer":{"name":"pip","version":"21.3.1"},"python":"3.7.12","implementation":{"name":"CPython","version":"3.7.12"},"distro":{"name":"Ubuntu","version":"20.04","id":"focal","libc":{"lib":"glibc","version":"2.31"}},"system":{"name":"Linux","release":"5.11.0-1021-azure"},"cpu":"x86_64","openssl_version":"OpenSSL 1.1.1f  31 Mar 2020","setuptools_version":"47.1.0"},"tls_protocol":"TLSv1.2","tls_cipher":"ECDHE-RSA-AES128-GCM-SHA256"}
{"timestamp":"2021-11-29 07:18:27 UTC","country_code":"US","url":"/packages/e3/1a/6affecd2344efee7f2487fac82242474cbac09f9e04929da5944907baf11/pytest_cov-2.11.1-py2.py3-none-any.whl","project":"pytest-cov","file":{"filename":"pytest_cov-2.11.1-py2.py3-none-any.whl","project":"pytest-cov","version":"2.11.1","type":"bdist_wheel"},"details":{"installer":{"name":"pip","version":"20.3.4"},"python":"2.7.13","implementation":{"name":"CPython","version":"2.7.13"},"distro":{"name":"Ubuntu","version":"20.04","id":"focal","libc":{"lib":"glibc","version":"2.31"}},"system":{"name":"Linux","release":"5.11.0-1021-aws"},"cpu":"x86_64","openssl_version":"OpenSSL 1.1.1f  31 Mar 2020","setuptools_version":"44.1.1"},"tls_protocol":"TLSv1.2","tls_cipher":"ECDHE-RSA-AES128-GCM-SHA256"}
{"timestamp":"2021-11-29 15:03:06 UTC","country_code":"US","url":"/packages/e3/1a/6affecd2344efee7f2487fac82242474cbac09f9e04929da5944907baf11/pytest_cov-2.11.1-py2.py3-none-any.whl","project":"pytest-cov","file":{"filename":"pytest_cov-2.11.1-py2.py3-none-any.whl","project":"pytest-cov","version":"2.11.1","type":"bdist_wheel"},"details":{"installer":{"name":"pip","version":"21.3.1"},"python":"3.8.12","implementation":{"name":"CPython","version":"3.8.12"},"distro":{"name":"Ubuntu","version":"20.04","id":"focal","libc":{"lib":"glibc","version":"2.31"}},"system":{"name":"Linux","release":"5.11.0-1021-azure"},"cpu":"x86_64","openssl_version":"OpenSSL 1.1.1f  31 Mar 2020","setuptools_version":"56.0.0"},"tls_protocol":"TLSv1.2","tls_cipher":"ECDHE-RSA-AES128-GCM-SHA256"}
{"timestamp":"2021-11-29 19:09:17 UTC","country_code":"US","url":"/packages/e3/1a/6affecd2344efee7f2487fac82242474cbac09f9e04929da5944907baf11/pytest_cov-2.11.1-py2.py3-none-any.whl","project":"pytest-cov","file":{"filename":"pytest_cov-2.11.1-py2.py3-none-any.whl","project":"pytest-cov","version":"2.11.1","type":"bdist_wheel"},"details":{"installer":{"name":"pip","version":"21.1.3"},"python":"3.7.11","implementation":{"name":"CPython","version":"3.7.11"},"distro":{"name":"Ubuntu","version":"20.04","id":"focal","libc":{"lib":"glibc","version":"2.31"}},"system":{"name":"Linux","release":"5.11.0-1020-aws"},"cpu":"x86_64","openssl_version":"OpenSSL 1.1.1l  24 Aug 2021","setuptools_version":"58.0.4"},"tls_protocol":"TLSv1.2","tls_cipher":"ECDHE-RSA-AES128-GCM-SHA256"}
{"timestamp":"2021-11-29 01:58:17 UTC","country_code":"US","url":"/packages/e3/1a/6affecd2344efee7f2487fac82242474cbac09f9e04929da5944907baf11/pytest_cov-2.11.1-py2.py3-none-any.whl","project":"pytest-cov","file":{"filename":"pytest_cov-2.11.1-py2.py3-none-any.whl","project":"pytest-cov","version":"2.11.1","type":"bdist_wheel"},"details":{"installer":{"name":"pip","version":"21.2.4"},"python":"3.6.15","implementation":{"name":"CPython","version":"3.6.15"},"distro":{"name":"Ubuntu","version":"20.04","id":"focal","libc":{"lib":"glibc","version":"2.31"}},"system":{"name":"Linux","release":"4.15.0-1115-aws"},"cpu":"x86_64","openssl_version":"OpenSSL 1.1.1f  31 Mar 2020","setuptools_version":"58.0.4"},"tls_protocol":"TLSv1.2","tls_cipher":"ECDHE-RSA-AES128-GCM-SHA256"}

Environment
  System:
    OS: macOS 11.6.1
    CPU: (8) arm64 Apple M1
    Memory: 141.98 MB / 8.00 GB
    Shell: 5.8 - /bin/zsh
  Binaries:
    Node: 16.13.0 - ~/.nvm/versions/node/v16.13.0/bin/node
    Yarn: 1.22.17 - /opt/homebrew/bin/yarn
    npm: 8.1.0 - ~/.nvm/versions/node/v16.13.0/bin/npm
  Managers:
    Homebrew: 3.3.7 - /opt/homebrew/bin/brew
    pip3: 21.1.3 - ~/.pyenv/shims/pip3
    RubyGems: 3.2.22 - /Users/ghaidar/.rbenv/shims/gem
  Utilities:
    CMake: 3.21.4 - /opt/homebrew/bin/cmake
    Make: 3.81 - /usr/bin/make
    GCC: 4.2.1 - /usr/bin/gcc
    Git: 2.34.1 - /opt/homebrew/bin/git
    Clang: 12.0.5 - /usr/bin/clang
  Servers:
    Apache: 2.4.48 - /usr/sbin/apachectl
  Virtualization:
    Docker: 20.10.8 - /usr/local/bin/docker
  IDEs:
    Nano: 2.0.6 - /usr/bin/nano
    VSCode: 1.62.3 - /usr/local/bin/code
    Vim: 8.2 - /usr/bin/vim
    Xcode: /undefined - /usr/bin/xcodebuild
  Languages:
    Bash: 3.2.57 - /bin/bash
    Go: 1.17.3 - /opt/homebrew/bin/go
    Perl: 5.30.2 - /usr/bin/perl
    PHP: 7.3.29 - /usr/bin/php
    Python: 3.9.6 - /Users/ghaidar/.pyenv/shims/python
    Python3: 3.9.6 - /Users/ghaidar/.pyenv/shims/python3
    Ruby: 3.0.2 - /Users/ghaidar/.rbenv/shims/ruby
  Databases:
    PostgreSQL: 11.13 - /opt/homebrew/bin/postgres
    SQLite: 3.32.3 - /usr/bin/sqlite3
  Browsers:
    Chrome: 96.0.4664.55
    Firefox: 94.0.2
    Safari: 15.1

Hey @disintegrator there's no back pressure here but it's because of the drop output rather than the broker or batching policies. The reason you're only noticing it with the batching enabled is likely because that's the bit that's allowing your pipeline to hit a throughput high enough to fill memory, otherwise the lower rate of message traversal provides just enough back pressure to keep within limits.
In order to force some kind of back pressure you could add a rate limit processor at the output level: https://www.benthos.dev/docs/components/processors/rate_limit, it'd look something like this:
output:
  broker:
    outputs:
      - drop: {}
        processors:
        - rate_limit:
            resource: drop_limit
    batching:
      period: 30s
      byte_size: 40_000_000

rate_limit_resources:
  - label: drop_limit
    local:
      count: 1000 # TODO: Tune this
      interval: 10ms

Or a simpler version would be to emulate a slow output with a sleep processor:
output:
  broker:
    outputs:
      - drop: {}
        processors:
        - sleep:
            duration: 10ms
    batching:
      period: 30s
      byte_size: 40_000_000

We should probably add a natural rate limit internal to the drop output to avoid these issues but I'm not sure what that limit should be tuned for.

@Jeffail Amazing! This is a facepalm moment. I did not notice the rate limit processor 😅
For what it's worth, the original config was writing the batches using the file output and the problem was the same. The config I shared is the minimum reproduction. I followed the docs advice of swapping bits out with drop to narrow down the issue.

@Jeffail I'm reopening this because I could not confirm that simulating a slow downstream output was causing backpressure. I tried an example from your advice and added a 1 hour sleep but my processor was quickly OOM-killed by docker:
input:
  file:
    paths: [${DATA_PATH}/input/pypi/*.jsonl]
    codec: lines
  processors:
    - bloblang: |
        map escape_csv {
          root = if this.re_match("[\"\n,]+") {
            "\"" + this.replace("\"", "\"\"") + "\""
          } else {
            this
          }
        }

        meta = deleted()

        root = [
          (this.timestamp | "").string().apply("escape_csv"),
          (this.project | "").string().apply("escape_csv"),
          (this.url | "").string().apply("escape_csv"),
          (this.details.installer.name | "").string().apply("escape_csv"),
          (this.details.installer.version | "").string().apply("escape_csv"),
          (this.details.implementation.name | "").string().apply("escape_csv"),
          (this.details.implementation.version | "").string().apply("escape_csv"),
          (this.details.system.name | "").string().apply("escape_csv"),
          (this.details.system.release | "").string().apply("escape_csv")
        ].join(",")
output:
  broker:
    outputs:
      - drop: {}
        processors:
          - sleep:
              duration: 60m
    batching:
      period: 1m
      byte_size: 40_000_000
      processors:
        - insert_part:
            index: 0
            content: "timestamp,project,url,installer_name,installer_version,implementation_name,implementation_version,system_name,system_release"
        - archive:
            format: lines
        - bloblang: |
            meta batch_hash = content().hash("xxhash64").encode("hex")

I suspect the batcher goroutine(?) is not waiting/suspending while downstreams process messages. Perhaps that is exactly in line with what you mentioned in your original comment:

The reason you're only noticing it with the batching enabled is likely because that's the bit that's allowing your pipeline to hit a throughput high enough to fill memory

If I understand this correctly then I think I'm making a feature request for a batching process that suspends until downstreams are ready in this issue. If that is the intended behaviour then I'll continue to investigate…

@disintegrator might be worth trying it with a reduced size as 40mb is very large relative to a 2gb limit, benthos messages are preserved internally in case of downstream errors so a 40mb batch will be buffered in potentially more than one form during its lifetime within the service. Each layer of components then adds one space for a message to be blocked waiting to progress, so I can imagine it's within the realm of possibility that you'd reach that limit even with back pressure working normally.

You're absolutely right. I just did a number of tests at different batch size with that 60m sleep and I was able to confirm that the memory held at stable point. Without the sleep my theory was that memory will fluctuate but not exceed the stable point and I am confident I confirmed this theory as well.
This all told me that backpressure was in effect and that I need to find what the stable point is for a 40MB batch size because it's not 2GB.
Thank you again for working with me on these questions!

@Jeffail You're absolutely right. I just did a number of tests at different batch size with that 60m sleep and I was able to confirm that the memory held at stable point. Without the sleep my theory was that memory will fluctuate but not exceed the stable point and I am confident I confirmed this theory as well.
This all told me that backpressure was in effect and that I need to find what the stable point is for a 40MB batch size because it's not 2GB.
Thank you again for working with me on these questions!

No worries, this is a bit of an extreme case but we really ought to be more memory efficient here. Unfortunately there's not much we can change internally yet until we mark v4 and can start refactoring internals.


Add more gradations to map zoom

Closes https://github.com/VEuPathDB/web-monorepo/issues/763

I set the value to 1000 and it worked for me on Chrome, Firefox, and Edge!


dns and node

2021-07-06T14:35:03.788 wallet kale.wallet.wallet_node    : WARNING  No keys present. Create keys with the UI, or with the 'kale keys' program.
2021-07-06T14:35:37.098 daemon asyncio                    : ERROR    Task exception was never retrieved
future: <Task finished name='Task-13' coro=<IocpProactor.accept..accept_coro() done, defined at asyncio\windows_events.py:563> exception=OSError(22, 'Il nome di rete specificato non è più disponibile', None, 64, None)>
Traceback (most recent call last):
File "asyncio\windows_events.py", line 566, in accept_coro
File "asyncio\windows_events.py", line 812, in _poll
File "asyncio\windows_events.py", line 555, in finish_accept
OSError: [WinError 64] Il nome di rete specificato non è più disponibile
2021-07-06T14:35:37.099 daemon asyncio                    : ERROR    Accept failed on a socket
socket: <asyncio.TransportSocket fd=824, family=AddressFamily.AF_INET6, type=SocketKind.SOCK_STREAM, proto=0, laddr=('::1', 56300, 0, 0)>
Traceback (most recent call last):
File "asyncio\proactor_events.py", line 818, in loop
File "asyncio\windows_events.py", line 566, in accept_coro
File "asyncio\windows_events.py", line 812, in _poll
File "asyncio\windows_events.py", line 555, in finish_accept
OSError: [WinError 64] Il nome di rete specificato non è più disponibile
2021-07-06T14:35:51.917 full_node kale.full_node.full_node: ERROR    Exception while querying DNS server: The DNS operation timed out after 30.003463745117188 seconds
2021-07-06T14:36:06.479 wallet kale.rpc.rpc_server        : WARNING  Cannot connect to daemon at ws://localhost:56300
2021-07-06T14:36:06.495 harvester kale.rpc.rpc_server     : WARNING  Cannot connect to daemon at ws://localhost:56300
2021-07-06T14:36:12.557 wallet kale.rpc.rpc_server        : WARNING  Cannot connect to daemon at ws://localhost:56300
2021-07-06T14:36:12.573 harvester kale.rpc.rpc_server     : WARNING  Cannot connect to daemon at ws://localhost:56300
2021-07-06T14:36:18.604 harvester kale.rpc.rpc_server     : WARNING  Cannot connect to daemon at ws://localhost:56300
2021-07-06T14:36:18.635 wallet kale.rpc.rpc_server        : WARNING  Cannot connect to daemon at ws://localhost:56300
2021-07-06T14:39:22.027 wallet kale.wallet.wallet_node    : WARNING  No keys present. Create keys with the UI, or with the 'kale keys' program.
2021-07-06T14:39:53.855 full_node kale.full_node.full_node: ERROR    Exception while querying DNS server: The DNS operation timed out after 30.000961780548096 seconds
2021-07-06T14:41:49.573 full_node kale.full_node.full_node: ERROR    Exception while querying DNS server: The DNS operation timed out after 30.014643669128418 seconds
how resolve this?

please try the new version.


TELEWORK: manager's approval or Rejection email message for submitter

As a BCPS employee, I want to receive a notification that my telework application has been rejected (so that I can get more information from the supervisor/delegate approver about my submission.
Potential rejection message:
"Hello (employee name),
Your Telework submission was not approved.  Please see the comments in the form or check with your Manager/Supervisor (name) or delegate who reviewed your submission.
Thank you."
ACCEPTANCE CRITERIA:
This ticket can be closed when:

the Telework pdf to employee contains a record/details
of the decision
we have a successful test the (correct) rejection message is sent to the employee
the test is successful in all three environments

@MeghanStothers Please review in TEST.

If this technical task is complete, let's close the ticket. Ten we can open a new one "email wording" so the writers can review/ere-write the message and ask form/owners (PIE) to sign off on the wording. Otherwise, this ticket will never be closed :)


Update default.yml with new RuboCop names

RuboCop renamed a few cops and removed FinalNewline

Looks like the build failed because the Gemfile has Rubocop 0.49.1 pinned and these changes require 0.51.0

Thanks for the PR!
Merged in f99766b4 with some follow ups in 99e6440c...1166057c.


Questions regarding ebucoreplus

Hi all, I got some questions regarding interpretation of classes and properties... I'm not sure if this is the right place to post them... let me know!
For example: http://www.ebu.ch/metadata/ontologies/ebucoreplus#Event
Looking at the subclasses, I suppose this should be interpreted more as an "occurence" in wikidata https://www.wikidata.org/wiki/Q1190554 than as an "event" https://www.wikidata.org/wiki/Q1656682
And there is no such class as a "planned" event yet (like festivals or a public broadcasting event)?
I might of course be missing or misunderstanding something :-)

I think the http://www.ebu.ch/metadata/ontologies/ebucoreplus#Event does not really distinguish between the two, i.e. it can be used for both. It has a start time and an end time and a location, which allows it to represent wikidata "occurences" https://www.wikidata.org/wiki/Q1190554.
And it can have an identifier, which probably makes it applicable also for wikidata "events" https://www.wikidata.org/wiki/Q1656682) and for "planned events" like a concert or a sports tournament.
Further questions are always welcome, but maybe their better placed in a list and directed towards the EBU-coordinator of the metadata modelling group (https://tech.ebu.ch/groups/mm)? Where would you expect to find the answers?
I could reuse the questions for a FAQ section in the documentation, eg. the wiki.

Hi @RendersKoen , please indicate whether the answer is satisfying or not. I could then close the issue. Thank you!

Yes thanks Juergen, excuse me for not having replied earlier!


Fix paths with spaces without introducing bash dependency

The ctags command fails on unix platforms when gutentags is installed in a path containing whitespace (or special characters for the shell).
This fixes that problem by supplying the options file as a separately quoted argument. Unlike #254, this does not introduce a bash dependency.
Fixes #237

Merged, thanks!

@ludovicchabant One thing to note about this is that if there are multiple -o arguments, only the last one takes effect. Currently it looks like it's possible that there are two -o arguments if there is a project options file and no file list command, in which case the recursive options file does not take effect. (This could also be considered a feature in the sense that if you have a project options file, then that is the only options file being used.)
A potential fix would be to pass the --recurse option directly instead of through a file, and use the -o only for the project options. Or perhaps introduce a new command-line argument which would be like -o except that it inlines the contents of the file, similar to #257…

This seems to have broken custom ctag executables.
Wrapping "$CTAGS_EXE" in double quotes breaks custom executables such as:
let g:gutentags_ctags_executable_ruby = 'ripper-tags --ignore-unsupported-options --recursive'

Wrapping "$CTAGS_EXE" in double quotes breaks custom executables such as:
let g:gutentags_ctags_executable_ruby = 'ripper-tags --ignore-unsupported-options --recursive'

Hmm, yes, on the other hand, not wrapping it in quotes breaks something like /opt/foo bar/bin/ctags. I would personally be in favor of having a separate option for custom arguments, rather than feeding them through the executable name.

Yeah we need quotes for the (weird :) ) people who put things in paths with spaces. If you have custom arguments to pass, you can do that with an options file, or with g:gutentags_ctags_extra_args.

Thanks g:gutentags_ctags_extra_args works!
let g:gutentags_ctags_executable_ruby = 'ripper-tags'
let g:gutentags_ctags_extra_args = ['--ignore-unsupported-options', '--recursive']

Re. paths with spaces, some operating systems have default paths with spaces, so it’s not always by choice.


[Feature] Open context menu when connection is dropped

Is your feature request related to a problem? Please describe.
I would like to open a context menu from a pending connection.
Describe the solution you'd like
When dropping a pending connection onto the editor's surface, a context menu should open showing all available nodes. When a node is selected, a new connection to that node should be created.
Additional context
Something similar to the context menu from UE4's Blueprint editor.

Example implementation in Calculator app


credential_process doesn't work for authentication

I've been getting bit on credential_process multiple places on products using the aws go sdk v2  (terraform,pulumi and now e1s 😄 etc) due to this bug I think you just might need to bump the sdk to the fixed version
 ~ ❯ e1s
sh: /Users/ryan.odonnell/go/bin/redacted: No such file or directory
e1s failed to start, please check your aws cli credential or permission.
operation error ECS: ListClusters, get identity: get credentials: failed to refresh cached credentials, process provider error: error in credential_process: exit status 127

Hi, I upgrade all aws deps to the latest. Please upgrade e1s to >v1.0.17.


Flow: Random seeding with bias does not work

To reproduce:

Load CM1/Orf/24Maycontrol and enable Flow. Streamlines are displayed
Change the seed distribution type to Random w/Bias. No streamlines are
drawn.

This bug is caused by an inconsistency between the GUI widget and the parameter data base.
More specifically, upon switching to Random w/ Bias mode, the GUI widget RakeBiasVariable displays a valid variable, while querying params->GetRakeBiasVariable() returns an empty string.

Assigning to @StasJ


UI iusse, Terminal window fixed and smaller than display.

On MacOS, the terminal window that launches is fixed in size, yet the display larger by about 20% making for scrolling the window in addition to scrolling the buffer.
Minor but annoying.

Also happens in Windows10

Appears to be a deliberate setting here
@toddtreece Was there a reason you locked down the terminal window size?


Timeline/Calendar View not working on iPad

What happened?
The Timeline/Calendar view on Mobile (specifically iPadOS) does not work. It works normally on my PC, but when I try to view it on iPad, it just does not render.
What did you expect to happen?
I expected for the Timeline/Calendar view to render properly like it does on a laptop, but it does not.

How can we reproduce it (as minimally and precisely as possible)?
Just attempt to render the Timeline/Calendar view on an iPad. If this does not reproduce the issue, here are the list of plugins that I currently have installed:

Anything else we need to know?
No response
Plugin version
1.17.4
Obsidian version
1.7.4 (167)
OS
Other

Thanks, this is a known issue that has been reported by #939. Sorry that didn't find time to shot a fix, testing on mobile is really painful. I'll close this thread as duplicate and please track dev progress there.


Make store and get in the storage library private

Simply remove the pub keyword from store and get here: https://github.com/FuelLabs/sway/blob/master/sway-lib-std/src/storage.sw. This is a breaking change that also requires documentation update. Also, as static arrays don't work in storage right now, this will be a bit limiting but that's probably fine in the short term as the benefits of not exposing store and get (which are basically footguns) outweigh the benefits of enabling reading and writing arrays.

If we are going to put every thing thatll ever have storage functionality in the storage.sw file, this is fine, if not this is not good

Agreed this is a short term solution until we have more flavours of pub.

Should any content that uses store & get be deleted entirely since it will no longer be usable?

There may be more libraries that need to do storage of custom structs. I dont think the benefits of privating store/get outweigh the losses. I dont think anyone is even trying to use these functions unless they know what they are doing?

There may be more libraries that need to do storage of custom structs. I dont think the benefits of privating store/get outweigh the losses. I dont think anyone is even trying to use these functions unless they know what they are doing?

I believe the sway libs repo uses these. It doesn't make sense to make them private because of worries of users using them incorrectly, it's akin to using an array index incorrectly... no point in nuking arrays from a language.
It's breaking, it has already been used and it exists in GH so people can look it up and copy/paste or recreate on their own. We also have it tested so it's better to maintain rather than apply the safety wheels to everything.

I do agree and I'm starting to lean toward not doing this. This is why this hasn't been done yet.
The good thing about store and get is that they provide a low level API that's pretty close to asm but not really asm. Users who want to manipulate storage slots manually (this should be rare) can use these instead of using asm blocks.
In light of the discussions above, I'm closing this issue.


Feat: Adding jump to top arrow in wishlist

Describe the feature
I had observed their is no jump to top arrow in wishlist.I want to add jump to top arrow in wishlist.
Add ScreenShots

Record

[X] I agree to follow this project's Code of Conduct
[X] I'm a GSSOC'24 contributor
[X] I want to work on this issue

Hello @harinisayani !
You can start working on this issue. Also, it seems that we cant go to wishlist form the homepage's navbar. Can you please look at the link and change it accordingly too?
Happy Coding! ✨

Same issue is mentioned in #591


Manually integrate beyond20

Since so many people use beyond20, it might be a good idea to manually trigger the beyond20 events (since some stuff currently doesn't work). Here is a link on how to do that:
https://beyond20.here-for-more.info/api#integrating-with-beyond20

This is now done


TokenExpiryTime for SMS OTP

Is your suggestion related to a missing or misleading document? Please describe.
TokenExpiryTime config property is not described in the doc [1]. It can be defined in the same way other properties defined. The unit is seconds.
Additionally, the following block has a typo. It has to be authentication.authenticator.sms_otp.parameters. s is missing.
[authentication.authenticator.sms_otp.parameter]
<Property-name> = <Property-value>

[1] https://is.docs.wso2.com/en/5.10.0/learn/configuring-sms-otp/#sms-otp-configuration-properties

This issue is being closed due to extended inactivity. Please feel free to reopen it if further attention is needed. Thank you for helping us keep the issue list relevant and focused!


Remove "Blazor Time Zone Kit"

Due to Blazor WebAssembly v.3.2 Preview 5 has been started support for time zones officially.
"Blazor Time Zone Kit" will no longer be maintained.
(I'm an owner of "Blazor Time Zone Kit".)

Ok I archive it. Thank's a lot.

Thanks!


在线 demo 打开报错

README 中提供的 demo 地址打开报错 https://65hzt.sse.codesandbox.io/
ℹ Compiling Webpack
[1] ✔ Webpack: Compiled with some errors in 36.61s
[1]  ERROR  Failed to compile with 3 errors10:27:34 PM
[1]
[1] These dependencies were not found:
[1]
[1] * fr-generator in ./packages/core/dist/core.esm.js
[1] * react-split-pane/lib/Pane in ./packages/core/dist/core.esm.js
[1] * react-split-pane/lib/SplitPane in ./packages/core/dist/core.esm.js
[1]
[1] To install them, you can run: npm install --save fr-generator react-split-pane/lib/Pane react-split-pane/lib/SplitPane
[1] (node:405) UnhandledPromiseRejectionWarning: AssertionError [ERR_ASSERTION]: chunk of umi not found.
[1]     at htmlChunks.forEach (/sandbox/node_modules/@umijs/preset-built-in/lib/plugins/commands/htmlUtils.js:104:27)
[1]     at Array.forEach (<anonymous>)
[1]     at chunksToFiles (/sandbox/node_modules/@umijs/preset-built-in/lib/plugins/commands/htmlUtils.js:93:14)
[1]     at /sandbox/node_modules/@umijs/preset-built-in/lib/plugins/commands/htmlUtils.js:188:32
[1]     at Generator.next (<anonymous>)
[1]     at asyncGeneratorStep (/sandbox/node_modules/@umijs/preset-built-in/lib/plugins/commands/htmlUtils.js:62:103)
[1]     at _next (/sandbox/node_modules/@umijs/preset-built-in/lib/plugins/commands/htmlUtils.js:64:194)
[1]     at process._tickCallback (internal/process/next_tick.js:68:7)
[1] (node:405) UnhandledPromiseRejectionWarning: Unhandledpromise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
[1] (node:405) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
[1] (node:405) UnhandledPromiseRejectionWarning: AssertionError [ERR_ASSERTION]: chunk of umi not found.
[1]     at htmlChunks.forEach (/sandbox/node_modules/@umijs/preset-built-in/lib/plugins/commands/htmlUtils.js:104:27)
[1]     at Array.forEach (<anonymous>)
[1]     at chunksToFiles (/sandbox/node_modules/@umijs/preset-built-in/lib/plugins/commands/htmlUtils.js:93:14)
[1]     at /sandbox/node_modules/@umijs/preset-built-in/lib/plugins/commands/htmlUtils.js:188:32
[1]     at Generator.next (<anonymous>)
[1]     at asyncGeneratorStep (/sandbox/node_modules/@umijs/preset-built-in/lib/plugins/commands/htmlUtils.js:62:103)
[1]     at _next (/sandbox/node_modules/@umijs/preset-built-in/lib/plugins/commands/htmlUtils.js:64:194)
[1]     at process._tickCallback (internal/process/next_tick.js:68:7)
[1] (node:405) UnhandledPromiseRejectionWarning: Unhandledpromise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 2)
[1] (node:405) UnhandledPromiseRejectionWarning: AssertionError [ERR_ASSERTION]: chunk of umi not found.
[1]     at htmlChunks.forEach (/sandbox/node_modules/@umijs/preset-built-in/lib/plugins/commands/htmlUtils.js:104:27)
[1]     at Array.forEach (<anonymous>)
[1]     at chunksToFiles (/sandbox/node_modules/@umijs/preset-built-in/lib/plugins/commands/htmlUtils.js:93:14)
[1]     at /sandbox/node_modules/@umijs/preset-built-in/lib/plugins/commands/htmlUtils.js:188:32
[1]     at Generator.next (<anonymous>)
[1]     at asyncGeneratorStep (/sandbox/node_modules/@umijs/preset-built-in/lib/plugins/commands/htmlUtils.js:62:103)
[1]     at _next (/sandbox/node_modules/@umijs/preset-built-in/lib/plugins/commands/htmlUtils.js:64:194)
[1]     at process._tickCallback (internal/process/next_tick.js:68:7)
[1] (node:405) UnhandledPromiseRejectionWarning: Unhandledpromise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 3)
[1] (node:405) UnhandledPromiseRejectionWarning: AssertionError [ERR_ASSERTION]: chunk of umi not found.
[1]     at htmlChunks.forEach (/sandbox/node_modules/@umijs/preset-built-in/lib/plugins/commands/htmlUtils.js:104:27)
[1]     at Array.forEach (<anonymous>)
[1]     at chunksToFiles (/sandbox/node_modules/@umijs/preset-built-in/lib/plugins/commands/htmlUtils.js:93:14)
[1]     at /sandbox/node_modules/@umijs/preset-built-in/lib/plugins/commands/htmlUtils.js:188:32
[1]     at Generator.next (<anonymous>)
[1]     at asyncGeneratorStep (/sandbox/node_modules/@umijs/preset-built-in/lib/plugins/commands/htmlUtils.js:62:103)
[1]     at _next (/sandbox/node_modules/@umijs/preset-built-in/lib/plugins/commands/htmlUtils.js:64:194)
[1]     at process._tickCallback (internal/process/next_tick.js:68:7)
[1] (node:405) UnhandledPromiseRejectionWarning: Unhandledpromise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 4)
[1] (node:405) UnhandledPromiseRejectionWarning: AssertionError [ERR_ASSERTION]: chunk of umi not found.
[1]     at htmlChunks.forEach (/sandbox/node_modules/@umijs/preset-built-in/lib/plugins/commands/htmlUtils.js:104:27)
[1]     at Array.forEach (<anonymous>)
[1]     at chunksToFiles (/sandbox/node_modules/@umijs/preset-built-in/lib/plugins/commands/htmlUtils.js:93:14)
[1]     at /sandbox/node_modules/@umijs/preset-built-in/lib/plugins/commands/htmlUtils.js:188:32
[1]     at Generator.next (<anonymous>)
[1]     at asyncGeneratorStep (/sandbox/node_modules/@umijs/preset-built-in/lib/plugins/commands/htmlUtils.js:62:103)
[1]     at _next (/sandbox/node_modules/@umijs/preset-built-in/lib/plugins/commands/htmlUtils.js:64:194)
[1]     at process._tickCallback (internal/process/next_tick.js:68:7)
[1] (node:405) UnhandledPromiseRejectionWarning: Unhandledpromise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 5)
[1] (node:405) UnhandledPromiseRejectionWarning: AssertionError [ERR_ASSERTION]: chunk of umi not found.
[1]     at htmlChunks.forEach (/sandbox/node_modules/@umijs/preset-built-in/lib/plugins/commands/htmlUtils.js:104:27)
[1]     at Array.forEach (<anonymous>)
[1]     at chunksToFiles (/sandbox/node_modules/@umijs/preset-built-in/lib/plugins/commands/htmlUtils.js:93:14)
[1]     at /sandbox/node_modules/@umijs/preset-built-in/lib/plugins/commands/htmlUtils.js:188:32
[1]     at Generator.next (<anonymous>)
[1]     at asyncGeneratorStep (/sandbox/node_modules/@umijs/preset-built-in/lib/plugins/commands/htmlUtils.js:62:103)
[1]     at _next (/sandbox/node_modules/@umijs/preset-built-in/lib/plugins/commands/htmlUtils.js:64:194)
[1]     at process._tickCallback (internal/process/next_tick.js:68:7)
[1] (node:405) UnhandledPromiseRejectionWarning: Unhandledpromise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 6)
[1] (node:405) UnhandledPromiseRejectionWarning: AssertionError [ERR_ASSERTION]: chunk of umi not found.
[1]     at htmlChunks.forEach (/sandbox/node_modules/@umijs/preset-built-in/lib/plugins/commands/htmlUtils.js:104:27)
[1]     at Array.forEach (<anonymous>)
[1]     at chunksToFiles (/sandbox/node_modules/@umijs/preset-built-in/lib/plugins/commands/htmlUtils.js:93:14)
[1]     at /sandbox/node_modules/@umijs/preset-built-in/lib/plugins/commands/htmlUtils.js:188:32
[1]     at Generator.next (<anonymous>)
[1]     at asyncGeneratorStep (/sandbox/node_modules/@umijs/preset-built-in/lib/plugins/commands/htmlUtils.js:62:103)
[1]     at _next (/sandbox/node_modules/@umijs/preset-built-in/lib/plugins/commands/htmlUtils.js:64:194)
[1]     at process._tickCallback (internal/process/next_tick.js:68:7)
[1] (node:405) UnhandledPromiseRejectionWarning: Unhandledpromise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 7)
[1] (node:405) UnhandledPromiseRejectionWarning: AssertionError [ERR_ASSERTION]: chunk of umi not found.
[1]     at htmlChunks.forEach (/sandbox/node_modules/@umijs/preset-built-in/lib/plugins/commands/htmlUtils.js:104:27)
[1]     at Array.forEach (<anonymous>)
[1]     at chunksToFiles (/sandbox/node_modules/@umijs/preset-built-in/lib/plugins/commands/htmlUtils.js:93:14)
[1]     at /sandbox/node_modules/@umijs/preset-built-in/lib/plugins/commands/htmlUtils.js:188:32
[1]     at Generator.next (<anonymous>)
[1]     at asyncGeneratorStep (/sandbox/node_modules/@umijs/preset-built-in/lib/plugins/commands/htmlUtils.js:62:103)
[1]     at _next (/sandbox/node_modules/@umijs/preset-built-in/lib/plugins/commands/htmlUtils.js:64:194)
[1]     at process._tickCallback (internal/process/next_tick.js:68:7)
[1] (node:405) UnhandledPromiseRejectionWarning: Unhandledpromise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 8)
[1] (node:405) UnhandledPromiseRejectionWarning: AssertionError [ERR_ASSERTION]: chunk of umi not found.
[1]     at htmlChunks.forEach (/sandbox/node_modules/@umijs/preset-built-in/lib/plugins/commands/htmlUtils.js:104:27)
[1]     at Array.forEach (<anonymous>)
[1]     at chunksToFiles (/sandbox/node_modules/@umijs/preset-built-in/lib/plugins/commands/htmlUtils.js:93:14)
[1]     at /sandbox/node_modules/@umijs/preset-built-in/lib/plugins/commands/htmlUtils.js:188:32
[1]     at Generator.next (<anonymous>)
[1]     at asyncGeneratorStep (/sandbox/node_modules/@umijs/preset-built-in/lib/plugins/commands/htmlUtils.js:62:103)
[1]     at _next (/sandbox/node_modules/@umijs/preset-built-in/lib/plugins/commands/htmlUtils.js:64:194)
[1]     at process._tickCallback (internal/process/next_tick.js:68:7)
[1] (node:405) UnhandledPromiseRejectionWarning: Unhandledpromise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 9)
[1] (node:405) UnhandledPromiseRejectionWarning: AssertionError [ERR_ASSERTION]: chunk of umi not found.
[1]     at htmlChunks.forEach (/sandbox/node_modules/@umijs/preset-built-in/lib/plugins/commands/htmlUtils.js:104:27)
[1]     at Array.forEach (<anonymous>)
[1]     at chunksToFiles (/sandbox/node_modules/@umijs/preset-built-in/lib/plugins/commands/htmlUtils.js:93:14)
[1]     at /sandbox/node_modules/@umijs/preset-built-in/lib/plugins/commands/htmlUtils.js:188:32
[1]     at Generator.next (<anonymous>)
[1]     at asyncGeneratorStep (/sandbox/node_modules/@umijs/preset-built-in/lib/plugins/commands/htmlUtils.js:62:103)
[1]     at _next (/sandbox/node_modules/@umijs/preset-built-in/lib/plugins/commands/htmlUtils.js:64:194)
[1]     at process._tickCallback (internal/process/next_tick.js:68:7)
[1] (node:405) UnhandledPromiseRejectionWarning: Unhandledpromise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 10)

已解决，可以访问线上地址：https://oyg3y.sse.codesandbox.io/

已解决，可以访问线上地址：https://oyg3y.sse.codesandbox.io/


Add files via upload

mansoura delta

@mkarem4 Please provide us with:

the school official website URL
the school street address, including city and country
an URL of a page on the official website where the school offers an IT-related long-term (>1 year) course
a proof, which shows that the school recognizes the domain you are submitting as an official email domain for the students.

Please provide us with a proof, which shows that the school recognizes the domain you are submitting as an official email domain for the students.
You can provide us with any of the following:

A copy of instruction (a hard-copy or electronic one) given to you by the school that tells you to use an email address in this email domain
An URL of a page at the official school website that says that students use this email domain
A reference to a recognized journal where an article was published by someone from the school, with explicit affiliation to the school, and e-mail address from the email domain
A WHOIS data that link the email domain to the school somehow
A copy of official letter from the school representative saying that the school provides e-mail addresses in this domain to the students
Anything else that can prove this fact in similar way, i.e. can be verified independently.

@mkarem4 You might miss my previous comment. Please react.

@mkarem4 I am sorry, this request remains unanswered for a while. I have to close it without merging. Please reopen and comment if you have an update regarding the aforementioned issue. Thank you for your understanding!


Update README - adding details of PARTHENOS

Adding details of the PARTHENOS website and training material.

Hello @padraic7a ,
Thank you for your pull request.
Don't you think this contribution would be better placed in the "User Guides" section?
If you agree, could you please make this modification?
Best regards

Thanks, @padraic7a !


Update README.md

when i extracted zip file i got a file your-archive.html and data folder.
when i renamed data folder to twitter-archive and paste it in the root of project folder and ran it i got an error

if this seems correct you can merge, else i have misread or didnt understood instructions.

ok senpai


Question about limiting concurrency

Our team is starting to learn fp-ts, and we are starting with some basic async examples (mostly pulled from here) Running a set of Tasks in sequence is great, but what is the idiomatic way to restrict concurrency? For example, Promise.map (in bluebird) allows you to set a concurrency limit like {concurrency: 4}. Maybe something like .sequence(task)(maxConcurrency)(tasks)
One solution might be to split the array into chunks, and then iterate the chunks, using sequence and flatMap. However, that would mean every Task in each chunk would have to complete before moving on to the next chunk - one long running task could hold up the whole operation.

You can take a look at this discussion: https://github.com/gcanti/fp-ts/issues/574

Ah, ok so p-map looks like the way to go. Thanks!


Fix panic in ARM builds of Prometheus

An extra sync.Pool was added during a refactor which caused some 64 bit,
atomically accessed variables to no longer be 64 bit aligned. By moving
all atomically accessed variables to the beginning of the struct they
are guaranteed to be 64 bit aligned.
Note that this PR is against the release-2.13 branch and will need to be
merged back into master.
cc: @krasi-georgiev since you are the release shepherd.
Closes #6105

I also found a couple places where this could happen in the remote code and applied the same fix/comment just in case.

:+1:

An extra sync.Pool was added during a refactor...

Which commit is this? I can't seem to find it.

@krasi-georgiev pull request #5999 (commit 3344bb5) in tsdb/head.go around line 67
https://github.com/prometheus/prometheus/pull/5999/files#diff-6fb13507bfadf9819fea5bda61f599e6L66

Thanks
Will wait few days and if there are no new bug reports will make a new 2.13.1 release.


Fix utils, use latest rest-client.

@rbellec I will look through this PR in the evening

Hello.
For the "byebug" this is really a mistake (I don't know how I let it in this commit) sorry.
About the amount, I understand (I also use integers only or Money gem for every money calculation), I just needed to have all test to green before using the gem for my client.  Ok to wait for the next 1.0 release. Thank for your answer !


Fix main build

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 62.67%. Comparing base (1ec111d) to head (62352d4).
Report is 28 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #4447      +/-   ##
==========================================
- Coverage   64.46%   62.67%   -1.79%
==========================================
  Files         586      586
  Lines       34201    37377    +3176
==========================================
+ Hits        22047    23426    +1379
- Misses      12154    13951    +1797

Flag
Coverage Δ

Debug
62.67% <ø> (-1.79%)
:arrow_down:

integration
?

production
62.67% <ø> (-1.79%)
:arrow_down:

unit
62.67% <ø> (-1.79%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.
see 188 files with indirect coverage changes


errors: implement (*Client).Close()

Currently there's no way to clean up an errors client. I want a service that reports errors on behalf of others (to do things like source mapping, etc). Not having a way to clean up means resources leak.

This is now a bigger requirement. We can stop all working Topics and make them flush before we Close.

@adams-sarah Could you take this one, too?

Yes, definitely.

(I should have commented on the wrong issue, ignore my comment above.)


regions version 0.2

This is an overview issue to have a place to discuss about scope, timescale, tasks for the regions version 0.2 release.
Here's the list of issues and pull requests with the 0.2 milestone:
https://github.com/astropy/regions/milestone/4
Of course this is up for debate, any pull request that's ready can go in, and we might put some tasks off to 0.3 if no-one has time to implement them.

@astrofrog made large contributions recently.
I'd like to make a regions 0.2 release soon.
Assigning myself here and putting Dec 5 (a week from now) as tentative release date.
There's no real blocker, but it would be nice to have the MPL plotting sorted out (#76, #77) and also the pytest-arraydiff testing (#80). @astrofrog - Do you have time for those within the next ~ week or should I start a pull request for one of those?

@cdeil - I have no time, because I have to focus on finishing up the WCSAxes PR to Astropy, so please feel free to go ahead with these changes! I can still review things, just no significant time to do.

With #85 merged, I think we are ready for the 0.2 release.
There's no more open issues / PR under the 0.2 milestone, several were moved to 0.3.
The motivation for the release is that we now have:

New: Mask, BoundingBox, see http://astropy-regions.readthedocs.io/en/latest/getting_started.html#masks
New: docs section on compound regions: http://astropy-regions.readthedocs.io/en/latest/getting_started.html#compounds
Improved: docs section on matplotlib plotting: http://astropy-regions.readthedocs.io/en/latest/getting_started.html#matplotlib-plotting and RectanglePixelRegion and EllipsePixelRegion have an as_patch method.
New: Start using Cython in this project, for mask generation
New: Start using pytest-arraydiff in this project, for automated region.to_mask testing.
Several small bug fixes and improvements, see here for the list of all pull requests merged between 0.1 and 0.2.

Am I missing something? I would put something like that in the announcement email on astropy-dev.
Maybe it would be simplest if we start a changelog and / or "what's new" sections, and just copy & paste those in the announcement emails?
Then the info would also be available in the package docs.
If there's something you'd like to get in, please file an issue or pull request in the next day or two.

I apologise for not making the release earlier as I said.
How about making the release now or next week?
Is anything except #103 by @larrybradley considered a must-have?
I would like to implement the things we discussed in January, but probably won't be able to take a day or two to do it, and I don't consider this a blocker.
Does anyone have plans to make pull requests in the coming ~ week or are you OK to make a v0.2 release now?

PyPI: https://pypi.python.org/pypi/regions/0.2
Tag: https://github.com/astropy/regions/releases/tag/v0.2
@astropy/regions-developers - Please try installing from the sdist on PyPI and comment here if everything looks good / tests pass for you.
I'll add 0.2 on RTD now.
@bsipocz - Do you have time to kick off the Conda build? Or should I do it?

There's conda install issues on RTD:
https://readthedocs.org/projects/astropy-regions/builds/5035533/
I'll try to fix that now by updating to newer versions.

Sure, I can do the conda update.

@bsipocz - Thanks!
I've updated the conda env on RTD in 5addcb9 in master.
Anyone know if that can be used to build the v0.2 docs on RTD?
If no -- is there a way I can fix the v0.2 docs build on RTD?

I guess you can set the label stable for that build, can't you?

I didn't manage to fix the docs build issues on RTD today.
I complained about them here
https://github.com/rtfd/readthedocs.org/issues/2614#issuecomment-280364651
and will try again in the coming days.

The last RTD docs build failed on conda install:
https://readthedocs.org/projects/astropy-regions/builds/5037252/
Next I tried with python 3.5 in 13886b7 and that worked for latest:
https://readthedocs.org/projects/astropy-regions/builds/5038871/
But for v0.2 the docs build fails on RTD, presumably because it's using the docs build setting from that version, not master:
https://readthedocs.org/projects/astropy-regions/builds/5038880/

I guess you can set the label stable for that build, can't you?

@bsipocz - I don't understand what the stable branch has to do with the v0.2 docs build!?
Anyone familiar with RTD knows how I can get the v0.2 docs build to work?
Is there an option that I do it locally and upload the result to RTD?

Macports package update: https://github.com/macports/macports-ports/pull/327
I'll wait for Mac / Linux conda packages to become available before announcing on astropy-dev:
https://github.com/astropy/conda-channel-astropy/pull/133#issuecomment-280575706

Here's a draft release announcement. Comment welcome.
One question I have: is this the recommended way to upgrade via pip, or should there be an extra flag that prevents updating of dependencies like Numpy or Astropy?

Hi all,
I am happy to announce the release of version 0.2 of the regions package.
The regions package features classes to represent pixel and sky regions with methods to do analysis (mainly containment check and mask generation) and matplotlib plotting, as well as DS9 region format serialisation.
Docs: http://astropy-regions.readthedocs.io
Code: https://github.com/astropy/regions
As usual, you can upgrade with pip or conda:
pip install  regions —upgrade
conda upgrade -c astropy regions
This regions package is in an early stage of development. It is not feature complete or API stable. Feedback or contributions highly welcome!
There is a tutorial introduction available that gives an overview of the current features and API here:
http://astropy-regions.readthedocs.io/en/latest/getting_started.html
The 0.2 release contains significant additions such as a PixCoord, BoundingBox and Mask class, as well as efficient circle and polygon pixel containment and mask code written in Cython, added by Tom Robitaille, Larry Bradley and Christoph Deil. Adam Ginsburg implemented several improvements and fixes to the DS9 region serialisation compared to v0.1.
The hope is that regions package development accelerates a bit and that we will reach a status where it’s appropriate to propose it for inclusion in the Astropy core as astropy.regions in time for the Astropy 2.0 release (i.e. by March or April 2017 latest).
Questions, feedback and contributions on this mailing list or on Github are very welcome!
Cheers,
Christoph

@cdeil  In general, I think we recommend using the --no-deps flag to prevent pip from unintentionally upgrading dependent packages (e.g. it's recommended in the photutils docs).

@cdeil - What I meant is that you can probably use the "stable" label on RTD and assign it to the build that passed (but was after the official 0.2 release). I haven't tried doing it though, so it's just guesswork. That case you have a stable and a latest RTD, which should be enough I think.

The conda packages are available now, so I suggest to send out the release announcement and close this issue/milestone.

Mailing list announcement:
https://groups.google.com/forum/#!topic/astropy-dev/uvFZL0zmBcA
I removed the part saying we'll try to finish this package by March or April and propose it to be included in Astropy core in time for 0.2.
This is not really a good idea or feasible at all, unless one or several people start working on it almost full-time for the coming weeks, and we get a lot of user feedback immediately.
So let's see how things go and where we are in 6 weeks.