HARP-6556: Generalize dynamic attributes for techniques.

Work-in-progress,
Support for general dynamic attributes in techniques (fade, interpolation, time) based on Expr

any attribute can be Expr in JSON
most of them are resolved to constitute unique Technique, but
some attributes are DynamicAttributes which can vary accross

features (like height, color), those affect geometry creation
time - time/frame/zoom/camera dependent material attributes

Splitted into #695, #697


utilities: also create latest directory

This commit makes create_site.sh create directory "latest",
containing the docs for the latest release.
This commit also fixes a bug.
Previously, create_site chooses to copy only files in directory
whose names begin with the current version.
Since the current version of google-cloud-java is 0.y.z,
libraries in alpha and beta are copied properly.
However, libraries in GA are not copied, because their versions
are x.y.z where x is not 0.

Changes Unknown when pulling a0f3786538ca2f8b21566eb145c042c4e845f5f7 on pongad:latest-doc into ** on GoogleCloudPlatform:master**.

Thank you @vam-google PTAL

Changes Unknown when pulling 08e4d5da6f6049424b9d662c0414e2bdff9dd15a on pongad:latest-doc into ** on GoogleCloudPlatform:master**.


Discussion to formalize 'project, observerandverifier` fields for localizations

The vars-localize and vars-gridview tools add additional information to an association's link_value json used to define the bounding box. These fields are:

observer - Who drew the boundingbox (using vars-localize)
verifier - Who confirmed the id of the bounding box (using vars-gridview)
project - info about what project the localizations were created for. e.g. FathomNet

When I merged the ML generated localizations into MBARI's main VARS database, I dropped these fields. So this issue is to track a discussion on how best to formalize the tracking of the data in these fields. I'd prefer to remove them from the JSON as it creates a lot of noise, but that might not be the best course of action. This should be a discussion with @kevinsbarnard and MBARI's video lab.

Project
My proposal is to drop the use of project completely. There's a column, observations.observation_group that should be used to track this. Currently, the way it's used is that primary annotations, those created by the video lab, are assigned to a group like ROV or i2MAP that groups them by platform type.
The ML annotations were imported into a group named ROV:training-set. These annotations might be redundant, essentially double counting observations. By putting them into a special group, they can be excluded from queries for quantitiative/qualitative data. Note that the other common group for ML annotations would be i2MAP:training-set.


Mobile Wallet. Online Send

Online Send operation might take time, up to 2 minutes especially if there are connection issues.
We need to:

show the progress is possible
Disable Send button until we get send confirmation.

Currently user can press send button several times and make several transactions instead of one.

@vinayaga07 ,  FYI, it is critical issue.  I will disable send while we are in the progress in Desktop app as well. In desktop it is less critical because progress exist.

@sarawut11 , assigning to you. The desktop part is done.

Verified with the latest build, hence closing this ticket.


JWT Cookie

First, sorry if I've got things wrong. This is my first project with feathersJS/react native...so I'have been struggling a bit, and trying to make things work.
I have been trying to get OAuth2 (Facebook) and Local authentication to work using the latest feathers-authentication(1.0.2)
I managed to get up to a point where I would see the accessToken being delivered as body of a GET request to the server, in the format as follows:
http://{my local dev server}/auth/facebook/callback?code={ a facebook generated code here...}#=
The accessToken, when decoded using JWT does include userID and the other properties. So, after I got this right, I tried to use it on the React Native App. The sample I found (https://github.com/sscaff1/hopePing) used CookieManager, for the simple reason that we cannot get the content of the response (inside the WebView) and we cannot run "fetch" on the same URL. I did not understand how to setup a "successRedirect"...so I went on to enable Cookies.
This is how my default.json looks like:
"auth": {
    "local":{},
    "cookie": {
    "enabled": true,
    "name": "feathers-jwt",
    "httpOnly": false,
    "secure": false,
        "domain": "my domain"
    },
    "jwt": {
        "header": { "typ": "access" },
        "audience": "my domain",
        "subject": "anonymous",
        "issuer": "feathers",
        "algorithm": "HS256",
        "expiresIn": "1d"
    },
    "session":false,
    "secret": " there is a secret here...",
    "facebook": {
        "clientID": "my facebook app ID",
        "clientSecret": "my facebook secret",
        "scope": ["public_profile","email"],
        "profileFields": ["id", "displayName", "email", "name", "cover", "picture"] //seems irrelevant
    },
    "google": {
        "clientID": "your google client id",
        "clientSecret": "your google client secret",
        "scope": ["profile"]
    }
}

But then,  the cookie would not show up on Chrome, no matter how I changed any of the above Cookie settings.
I set the Debug=feathers-authentication* and then I realized the last log I got for the Set-Cookie.js was
debug('Running setCookie middleware with options:', options);
I've put a temporary log after:
if (options.enabled && options.name) {
and it would show.
I pinpointed that the following condition was not met:
if (res.hook && res.hook.method !== 'remove' && res.data && res.data.accessToken) {
res.hook was undefined. If I remove res.hook && res.hook.method!=='remove', it works as expected, but I get 2 cookies for the same name/content.
Sorry for the long text, I hope I have explained everything properly. I wanted to know if I'm doing something wrong.
thanks in advance.

On a side note, I wanted to ask if the Authentication 1.0.2 should populate the Users' table with more than the {provider}Id column. Currently I can only see the that column getting populated while the others are null.
I'm using mySQL and Sequelize to back up the Users service.

@muvhaus-sl I'm not experienced with React Native, yet.  Last month we created the feathers-authentication-popups module that helps handle the auth workflow for web apps.  We've discussed how to use it for React Native, but we have yet to implement it.  It's one of the very next things on the list, though.

The reason that it only adds providerId is that we had problems with databases that require the data format to be pre-defined (e.g. all *SQL dbs). In that case you'll have to map the information you want from the oAuth callback to the fields you defined anyway.

@daffl ,
Is there any example that I can refer to?
Sorry if I'm being too demanding. It has been a very steep learning curve to get where I am, hopefully I can gain understanding and contribute back soon enough.

@marshallswain
On react native, the CookieManager setup that has been mentioned a few times on other issues works fine. I got Facebook to work fair enough.
So far, only Google has blocked the usage of Webviews for OAuth and I don't think its for technical reasons, I think it has been just another corporate move...
Perhaps the viable solution would be for the feathers client to have a widget like Auth0's lock? However, I must say, its a lot of trouble and I think it is not part of the scope of what Feathers is trying to do?
I don't think there is a clean way of doing a client side "widget" if OAuth providers start having different requirements...

experiencing the same issue as in the original question. In my case, I'm trying to migrate the 0.7.x /auth/local route to using /login, as described by the line in the README.
app.post('/login', auth.express.authenticate('local', { successRedirect: '/app', failureRedirect: '/login' }));

The redirect part works. However, after trying for a couple hours I too couldn't get the authToken to show up in the cookie after redirect. Tried manually setting auth.express.setCookie as a middleware to the post request and passing in app.get('auth') but to no avail. Putting a logger into the actual setCookie middleware revealed that it was never being called during a "normal" login flow where a user posts to /login.
All of that to say, I'm pretty confused

@jiangts,
Do you have this on your configuration:
"cookie": {
    "enabled": true,
    "name": "feathers-jwt",
    "httpOnly": false,
    "secure": false,  //this is not very safe for production usage, but I dont have https for testing.
        "domain": "my domain" //replace with your actual domain or IP address
    },

Then, the other thing I did to find out the reasons was to set the debug flag for "feathers-authentication*"
In doing that, I could spot where the setCookie was stopping.

Hi @muvhaus-sl,
I followed your suggestion and got the following log:
feathers-authentication-local:verify Checking credentials +22ms <email> <password>
  feathers-authentication-local:verify user found +13ms
  feathers-authentication-local:verify Verifying password +1ms
  feathers-authentication-local:verify Password correct +205ms
  feathers-authentication:passport:authenticate 'local' authentication strategy succeeded +0ms { _id: 586d446040ccc98452435a84,
  email: '<email>',
  password: '<pw hash>',
  __v: 0,
  updatedAt: 2017-01-04T18:52:16.385Z,
  createdAt: 2017-01-04T18:52:16.385Z } { userId: 586d446040ccc98452435a84 }
  feathers-authentication:express:authenticate Redirecting to / +1ms
  feathers-authentication:express:expose-headers Exposing Express headers to hooks and services +4s
  feathers-authentication:express:expose-cookies Exposing Express cookies to hooks and services +0ms undefined

Still not sure where setCookie failed...
Did you get something similar?

@jiangts
How is your  auth config looking like? And, do the Cookie domain and the actual host name match?
I would assume that if your cookie domain is set to "something.com" and you are accessing the server as "<IP_ADDRESS>" or "localhost", then the browser will not get the cookie?
I did not use Express, so the Expose-cookies is not triggered. On my case it uses Set-cookies.

@muvhaus-sl
I figured it out. However, I haven't gotten a flow that's as easy/simple as the version in 0.7.x with both setting the cookie and also redirecting the user. In fact, I had to write quite a bit of code on my own instead of just "glue" code over feathers-auth. I'm not sure if this is intentional for such a common use case (just a normal login), or maybe I'm missing something, but in either case, this wasn't anything like pleasant experience I've had so far with feathers :(
For future people, my solution was to create a new service called /login and put auth.express.setCookie(app.get('auth')) as a middleware to that service. This gets the cookie set properly. Then, I wrote some of my own code to do the redirect and clearing the cookie in case of auth failure, as auth.express.successRedirect and auth.express.failureRedirect didn't quite do what I wanted.

So @marshallswain can probably explain this much better than me but the problem of allowing the JWT in a cookie for all API endpoints (and why it has been removed) is that the API will become vulnerable to Cross Site Request Forgery (CSRF), so if someone hijacks a users browser they can make any authorized requests to the API. This is why only specific routes should be explicitly enabled for cookies. Maybe @ekryski can weight in if there is something else that can be done better for your usecase though.

@muvhaus-sl @jiangts can you post a link to a github repo? Quite frankly, there might be a bug in setting the cookie but it also might be that you just have things set up a different way. @jiangts the intention is for it to work the same as 0.7.x but there are a few more scenarios that we are covering now so we have those extra guards in the set-cookie middleware.

I also was having a similar issue. Perhaps I am misunderstanding something. One of the purposes of enabling cookies is to have non-SPAs (old school apps) be able to authenticate without having to set up tokens in the header, correct? Otherwise I don't see the purpose of the cookie being saved in the browser and resent along with every request. And yet you say that due to CSRF cookies are disabled on all routes (if I'm understanding you.) I thought that enabling the cookie option in the config allows cookies on all routes, and the authentication schemes/providers should be able to read it like normal.
When I use the provided middlewhere, a cookie does NOT get set (both cookies and sessions have been enabled, confirmed via debugging):
app.post('/login', auth.express.authenticate('local', { 'successLogin: '/', failureRedirect: '/login' }));

I have also tried something like the following:
app.post('/login', function(req, res, next) {
  return app.service('authentication').create({
    email<EMAIL_ADDRESS>    password: 'asdfasdf'
  }).then(function (result) {
    console.log('Authenticated with result: ', result);
    next();
  })
});

Additionally, I've tried setting up the setCookie middleware manually to no avail:
app.post('/login', auth.express.authenticate('local', { failureRedirect: '/' })); // note, no successRedirect, to get to the cookie middleware
app.post('/login', auth.express.setCookie(app.get('auth')));
app.post('/login', function (req, res) {
  res.send('DONE!');
});

If I do a POST request directly to the authentication service endpoint, I get a token AND a cookie set properly, so I know cookies are being set OK, just not with regular express routes.
Further digging shows that the setCookie middleware gets to this point:
if (res.hook && res.hook.method !== 'remove' && res.data && res.data.accessToken) {```

There is no hook property on the response, and this is where I'm stuck -- I can't figure out where hook gets added to the response. I've been at this the entire day, and I don't know what else to try.
My apologies if this should be obvious, I'm quite new to Express, let alone feathers. But nowhere in the documentation do I see where hook gets set, nor how cookies can be used with regular routes.
Thanks in advance.

@eblanshey
My original issue had the same symptom as yours: set-cookie.js would stop at the condition.
res.hook
With res.hook being undefined.
For a speedy solution, I commented that part of the condition and it looks like everything is working as expected. I'm also not sure why it was undefined to start with.
If I understood it right, on the
auth.hooks.authenticate(['jwt', 'local'])
You need to include 'cookie' on that array. I had to include "facebook" if I wanted to allow facebook authentication, for instance. And as far as I understand, cookie becomes a strategy if enabled, so you need to allow it on the list of possible "authenticate" strategies.
Looking forward for a more elegant solution on the set-cookie, as well as some corrections from the more experienced people when it comes to the setup.

@muvhaus-sl unfortunately adding "cookie" or "cookies" to the array doesn't work: Unknown authentication strategy 'cookie'. I think the functionality is simply broken.
I did a quick fix by adding a middleware that adds the Authorization header from the value of the cookie (using cookie-parser library):
app.use(function(req, res, next) {
  let cookieValue = req.cookies[app.get('auth').cookie.name];
  if (typeof cookieValue === 'string') {
    req.headers.authorization = "Bearer "+cookieValue;
  }

  next();
});

To the core devs: I understand feathers is mainly geared towards creating APIs, but full support and documentation for "old school" apps would great, especially since the docs guide you through how to set up view engines and what not.
My use case for using feathers is to quickly create a server-side rendered app, under the hood using the services to fetch data. That way I'd have a fairly easy roadmap to changing it to an SPA down the road when I have more time (just using the service endpoints that the traditional routes use). This could be a great selling point for devs to use Feathers for their next project. As it stands now, authentication using traditional methods is very difficult to understand. I suppose adding built-in CSRF protection would be needed, which could be a matter of using https://github.com/expressjs/csurf and automatically applying it to all non-service routes.

I believe I have got the same issue as seeing the following debug statement on my feathers server, and my cookies are not being set following a successful authentication:
feathers-authentication:express:expose-cookies Exposing Express cookies to hooks and services +0ms undefined

My auth config has the following set for cookies:
"cookie": {
    "enabled": true,
    "name": "feathers-jwt",
    "httpOnly": false,
    "secure": false
}

@eblanshey

One of the purposes of enabling cookies is to have non-SPAs (old school apps) be able to authenticate without having to set up tokens in the header, correct?  Otherwise I don't see the purpose of the cookie being saved in the browser and resent along with every request. And yet you say that due to CSRF cookies are disabled on all routes (if I'm understanding you.) I thought that enabling the cookie option in the config allows cookies on all routes, and the authentication schemes/providers should be able to read it like normal.

I'm really short on time, today, so this is going to kind of come out as a stream of consciousness, but I'll do my best.
It's pretty common to conflate enabling cookies with turning on "old-school" auth for routes.  With Feathers auth we separate the two.  Turning cookies on in the config just creates a cookie.  It doesn't automatically enable old school, CSRF-vulnerable routes.
It's important to keep the SSR server and API server as distinct entities.  The SSR server can pull the JWT from the cookie, decode it to get the payload.  This will include whatever entity data you've populated inside the JWT (like userId).  You can then use this data inside the params
import decode from 'jwt-decode';

app.use(function(req, res, next) {
  let jwt = req.cookies[app.get('auth').cookie.name];
  let payload = decode(jwt);
  // Only use service.find and service.get for GET requests
  app.service('todos').get({query: {}, user: payload.userId})
    .then(response => {
       // Use the response data and assemble your HTML for the response.
    });
});

You probably don't want to expose that cookie value to be used by the rest of the API server.  At least, I wouldn't want to deal with CSRF at all.  Even a good CSRF library is only going to mitigate CSRF attacks, not eliminate them.  If you keep the cookie auth data away from the API server, you've eliminated the risk of CSRF completely.  If your goal is to enable SSR it's just not necessary to expose cookie auth data to the API server.  The example above works more like an auth proxy to get authenticated data from the server in order to form HTML content.  (You'd want to make sure CORS is disabled for SSR endpoints)
We chose to keep the API server distanced from cookie auth because most people aren't aware of the risks or how to mitigate them.  And in most cases I've seen where people want to turn cookie support on, there's an alternative method that doesn't introduce security or privacy issues.  From my perspective, an API server should probably only expect a cookie for auth as an additional check on top of requiring a header.  Otherwise, API servers shouldn't be able to consume auth data from a cookie.

Running into the same issue as well upgrading a React Native app from feathers-auth 0.7.x to 1.0.  The app uses the cookie from an OAuth webview flow purely to obtain and store the JWT.  Subsequent calls to services use sockets and bearer token authentication.
Is it possible to have the cookie set just for the /auth/{provider} request and subsequent redirect?

I'll take a look at this today. There must be a bug somewhere.

For myself, a working chat app update would answer many questions.  Using the standard chat app from the Feathers book, I attempted to update to auth 1.0.2 and related packages using the migration guide.
Here is my services/authentication/index.js:
'use strict';
const auth = require('feathers-authentication');
const local = require('feathers-authentication-local');
const jwt = require('feathers-authentication-jwt');
module.exports = function() {
const app = this;
let config = app.get('auth');
app.configure(auth(config))
.configure(jwt())
.configure(local())
app.service('authentication')
.hooks({
before: {
// You can chain multiple strategies on create method
create: auth.hooks.authenticate(['jwt', 'local'])
}
});
};
I get this in the body when I authenticate:
{"accessToken":"eyJhbGciOiJIUzUxMiIsInR5cCI6ImFjY2VzcyJ9.eyJ1c2VySWQiOiJOaUtlb3E2cTQ4cU1Sa0paIiwiaWF0IjoxNDg2NTg2NzgzLCJleHAiOjE0ODkxNzg3ODMsImF1ZCI6Imh0dHBzOi8vMTI3LjAuMC4xIiwiaXNzIjoiZmVhdGhlcnMiLCJzaXYBOiJhbm9ueW1vdXMifQ.tTYovuN39PMJxCmK6-I_vX9-LP6vKSa7Gu4ttYAZioD0ifiSOamcmTs4hz3nlPGu5wGBkTTtNSZDuPrOxKNTKQ"}

For myself, a working chat app update would answer many questions. Using the standard chat app from the Feathers book, I attempted to update to auth 1.0.2 and related packages using the migration guide.
Here is my services/authentication/index.js:

const auth = require('feathers-authentication');
const local = require('feathers-authentication-local');
const jwt = require('feathers-authentication-jwt');

module.exports = function() {
  const app = this;

  let config = app.get('auth');

  app.configure(auth(config))
    .configure(jwt())
    .configure(local())

  app.service('authentication')
    .hooks({
      before: {
        // You can chain multiple strategies on create method
        create: auth.hooks.authenticate(['jwt', 'local'])
      }
    });
};

I get this in the body when I authenticate:
{"accessToken":"eyJhbGciOiJIUzUxMiIsInR5cCI6ImFjY2VzcyJ9.eyJ1c2VySWQiOiJOaUtlb3E2cTQ4cU1Sa0paIiwiaWF0IjoxNDg2NTg2NzgzLCJleHAiOjE0ODkxNzg3ODMsImF1ZCI6Imh0dHBzOi8vMTI3LjAuMC4xIiwiaXNzIjoiZmVhdGhlcnMiLCJzaXYBOiJhbm9ueW1vdXMifQ.tTYovuN39PMJxCmK6-I_vX9-LP6vKSa7Gu4ttYAZioD0ifiSOamcmTs4hz3nlPGu5wGBkTTtNSZDuPrOxKNTKQ"}

tldr; I can get token, what do I do with it to validate the logged in user on subsequent routes/navigations?
I'm authenticating users with front-facing login form /, then redirecting to a main home page /main.
My set up is a little complicated (MySQL + Feathers + VueJS SSR (really pre-rendering, but I haven't gotten that to work so I'm just using express-vue) + Angular), but that shouldn't matter much.
I can use the feathersClient to authenticate users, but I don't know what to do with the response. I can authenticate, but then if I manually go to a page that is behind an auth check it fails. I assume this is because cookie check fails?
This is my default.json's auth property.
  "auth": {
    "secret": "...bigsecret...",
    "strategies": [
      "local",
      "jwt"
    ],
    "path": "/authentication",
    "service": "users",
    "jwt": {
      "header": {
        "type": "access"
      },
      "audience": "localhost", // TODO probably need to programatically get this...?
      "subject": "anonymous",
      "issuer": "feathers",
      "algorithm": "HS256",
      "expiresIn": "1d"
    },
    "local": {
      "entity": "users",
      "service": "users",
      "usernameField": "username",
      "passwordField": "password"
    },
    "cookie": {
      "enabled": true,
      "domain": "localhost",
      "name": "feathers-jwt",
      "httpOnly": true,
      "secure": false // TODO make sure this is true in prod
    }
  }

Here is my authentication service and hooks
const authentication = require('feathers-authentication');
const jwt = require('feathers-authentication-jwt');
const local = require('feathers-authentication-local');
const manage = require('feathers-authentication-management');

module.exports = function() {
  const app = this;
  const config = app.get('auth');
  const managedConfig = app.get('managedAuth');

  // Set up authentication with the secret
  app.configure(authentication(config));
  app.configure(jwt());
  app.configure(local(config.local));
  app.configure(manage(managedConfig));

  app.service('authentication').hooks({
    before: {
      create: [
        authentication.hooks.authenticate(config.strategies)
      ],
      remove: [
        authentication.hooks.authenticate('jwt')
      ]
    }
  });
}

From the client's perspective, am I supposed to be able to use feathersClient to authenticate and then future navigation just works? Because this is not so 😢
I add routes with a function like so (I can just pass this to a forEach over an array of routes, makes it trivially easy to add new routes)
  let buildRoute = r => {
    app.use(
      '/' + r.path,
      auth.express.authenticate('jwt'),
      (req, res) => {
        res.render('main', routeData[r.name]);
      }
    );
  };

Ideally, wouldn't the feathers server set the header for the client? I'm at a loss of what I'm supposed to do...
To repro - set up normal feathers app, latest authentication - I'm using username and password as opposed to email (and only that, no OAuth for this project) - client landing page is / which is login form (don't even need to set up html, can just use chrome debugger), then use feathers client to get back the token. Clicking a link to /main or manually navigating to /main or even location.href= /main all don't work, even though the server should know this user is authenticated. I don't even know if this is supposed to work though...
I think I'm just missing something though... This should be easy, right? Logging users in? 😭

Is there any update on the topic?
I see it started from the react native, but I use regular react and have the issue that after the auth any cookie is set.
https://github.com/feathersjs/feathers-authentication-local/issues/17#issuecomment-302887815
not sure if the issue is the same, but I assume so.
Do I understand right, that the use case of the cookie jwt is, that after the page is refreshed manually - customer doesn't need to authenticate himself again?
Regards,

If you're going to use the feathers client with Socket.io to make requests, authentication has to happen on every refresh.  This is a limitation of the socket auth.

@marshallswain
yes, thanks, but if the cookie is set and the token is still valid, the authentication could be more or less "automatic".
therefore the fact of setting the cookie automatically is important.

I also tracked that this piece of code:
https://github.com/feathersjs/feathers-authentication/blob/master/src/express/set-cookie.js#L14
is never called (in my case at least), however the setCookie is registered.

Correct.  We left it out on purpose to avoid CSRF attacks.  You have to manually register it.  The cookie is still useful in SSR scenarios without that middleware enabled, so it's not enabled automatically when you turn on the cookie option.

Ah that's the case.
@marshallswain could you, please, confirm that I understood it right.
The feathers-jwt cookie would be set (on the client) after the correct authentication only in case I register the express middleware explicitly? Like it's described here: https://docs.feathersjs.com/api/authentication/server.html#express-middleware .
And the option cookie: {enabled: true} is not enough?

Oops.  I meant the expose-cookie middleware.  set-cookie should run with the options you posted.  On a development machine, you'll need these options to get a cookie:
    enabled: false, // whether the cookie should be enabled
    name: 'feathers-jwt', // the cookie name
    httpOnly: false, // whether the cookie should not be available to client side JavaScript
    secure: false // whether cookies should only be available over HTTPS

That middleware should run on successful authentication.  Do you have your before hooks on the authentication service setup?

@marshallswain
yes, I think I have everything set:
https://github.com/PavelPolyakov/planning-poker/blob/feature/authentication/src/server/services/authentication/index.js
here I describe more about the issue:
https://github.com/feathersjs/feathers-authentication-local/issues/17#issuecomment-302887815
Regards,

My experience to fix this kind of issue. It may help => #541:


@conda-forge-admin, please restart ci

Additional comment:
No response

@conda-forge-admin, please restart ci


Is there any implementation of a pager?

ag has a pager option so you can use ag --pager=less keyword to use less to browser all matches. Is this implemented in pt?

I would like to know about this as well

What's wrong with piping the result of pt through the pager of your choice?
pt ... | less

@StefanSchroeder The coloring (of matched text) will be dropped if you just pipe the result into less.

@district10 Hi, I recommend you use piping.
If you want color and grouping into less, you can specify --color, --group option.
Their options always coloring, grouping.
pt --color --group PATTERN | less

@monochromegane It works! Thank you.


sed -s /staking/delegation/

Not only would this make Duncan happy, but it's also much clearer and aligned with the specs :)

I assume this changes external API?


Add RSS feed for stories.

Summary
Add feed link in the header.
Also format the RSS feed output as an image.
Relevant Technical Choices
To repeated code, I moved the logic to render in feed for stories to a trait, so we can reuse. It isn't the cleanest thing, but it works.
I looked at enabling feeds with adding
'rewrite' => [
    'feeds' => true,
]

But this didn't work without enabling archives.
I also moved the logic for render the story in feed, into its own file.
To-do

User-facing changes

Testing Instructions

Fixes #3136

@swissspidy Can you re-review.


Comick Chapter Id not found

Source information
Comick 1.4.47
Source language
English
Steps to reproduce

Filter scan groups
Reset filter
Choose chapter from different scan

Expected behavior
Read chapter like normal
Actual behavior

Mihon/Tachiyomi version
Yokai <IP_ADDRESS>
Android version
Android 14
Other details
No response
Acknowledgements

[X] I have searched the existing issues and this is a new ticket, NOT a duplicate or related to another open or closed issue.
[X] I have written a short but informative title.
[X] I have updated the app to version 0.15.3.
[X] I have updated all installed extensions.
[X] I have tried the troubleshooting guide.
[X] If this is an issue with the app itself, I should be opening an issue in the app repository.
[X] I will fill out all of the requested information in this form.

That's an app-related issue since the app is the one handling the filtering (and it works fine on Mihon), check with Yokai GitHub instead


cannot use frontMatter as the name option

Hello, when using this plugin, I cannot get the content of frontmatter with the default setup.
However, if I do:
remarkPlugins: [remarkFrontmatter, [remarkMdxFrontmatter, {name: 'matter'}]]

I can get the content by file.matter. the custom name can be anything except for frontMatter (I'll get a undefined object)
I think this might be a bug.
See discussion here: https://stackoverflow.com/a/76962139/5737534

I don’t understand what you’re trying to do and what’s not working. Can you create a minimal runnable example to demonstrate the problem? Or better: Could you create a pull request with a failing test case?

I'm trying to get the content of the frontmatter.
I can only get it by rename the name to a keyword that is not frontmatter (can be matter, dog, cat, etc.)

Let me help you formulate the problem:
Given the following script:

```js
import { compile } from '@mdx-js/mdx'
import remarkFrontmatter from 'remark-frontmatter'
import remarkMdxFrontmatter from 'remark-mdx-frontmatter'

const mdx = `---
foo: bar
---

Some content
`

const file = await compile(mdx, {
  remarkPlugins: [
    remarkFrontmatter,
    remarkMdxFrontmatter
  ]
})

console.log(String(file))
```

I expected the following result:

```js

```

But instead I got this:

```js

```

I used the following package versions:

- `@mdx-js/mdx`:
- `remark-frontmatter`:
- `remark-mdx-frontmatter`:


Replace podspec scripting with standalone pod

We're currently fetching the xcframework and source files from https://github.com/matrix-org/matrix-rust-components-swift via a prepare_command script. It would be better to encapsulate this into a pod of its own (possibly even on the upstream repo).

This is obsolete now that we're using uniffi-bindgen-react-native.


(docs): increase readability

Update tense
Correct spelling
Update grammar
Update punctuation

Thanks for your contribution! I'm going to merge this PR into a rollup branch so we can merge the smaller changes into master in one go.


Pull requests

I added the .mds to the project, fixed the image links in the .mds and added the link to my first kata solution.
Regards

Hi there! I sincerely apologize for the super late merge. Long story short, I wanted to do it much earlier, but got caught up with something, and completely forgot about it. So... very sorry, and thank you so much!


Challenges/Lessons learned of Teachers currently teaching RSE things

1a: Challenges/Lessons learned of Teachers currently teaching RSE things
1b: Who tries Challenges/Lessoes to incorporate RSE topics in their teaching?

Some points to get started:
Big picture

Teaching a dedicated course -vs- injecting elements in domain courses

Maybe injecting elements in domain courses reaches out to more people?
Teaching a dedicated course to a heterogeneous audience may make it difficult to connect to real use cases and problems?

Lecture, Seminar, or Practical course?

Lecture: More content, content needs to be maintained, little practice opportunities
Seminar: More broad overview, students update the content, needs supervision
Practical course: Less content, deeper practice, needs supervision and maintenance

Lecture + exercises and/or lecture + projects

How easy is it to add projects to existing courses? ECTS limits?

Reviews and grading

How to do code reviews?
How to grade?
Methods and tools for doing peer-reviews? To what extent is this legal?
How to check plagiarism? How to check for automatically-generated solutions? How to design exercises that a human cannot just solve (correctly) by asking ChatGPT?

Infrastructure

We need (in some cases) to teach Linux-related tools, but students only know and have Windows installed. Do we adapt to their system (also offer Windows alternatives), or do we require them to find a solution (e.g., WSL)?
What to distribute to students that don't have Linux installed?

Instructions for WSL
A self-contained VM
A self-contained Docker image

Content

What is trivial, what is too advanced?
What do students actually find new and enjoy? What have they already seen 100 times?

Teamwork

How large a team is appropriate for each project?
How to ensure everyone is actively contributing?

To be continued

...

What to distribute to students that don't have Linux installed?

Instructions for WSL

I did that once in a lecture for biologists and it worked somewhat okay, but it also was a small course.

There is quite an overlap with #227. The big picture stuff should be covered by that issue.

There is quite an overlap with #227. The big picture stuff should be covered by that issue.

I see the overlap, but I think we can also take different viewpoints here. I mainly meant it as "what has the best learning/investment ratio for me as a teacher". But definitely, I will move that there as well.

I would also ask about who is the audience. Are RSE skills taught to SE who want to move to RSE or to non-SE (or CS or similar) who have done some development and want to switch to RSE?

Summary from the world-café discussion in deRSE24
@CaptainSifff should I put this text somewhere else in the repository?
pad
Participants of the deRSE24 conference in Würzburg discussed current challenges and lessons learned
from teaching RSE topics. In this discussion, teaching activities spanned from teaching university
courses, to daily mentoring junior professionals.
Key challenges that arose revolved around communication and interdisciplinarity.
As RSE students and mentees typically come from various home domains, it is
often challenging to estimate the level of previous knowledge, the needed learning outcomes, and
the terminology that each audience will be familiar with. In team projects, the teams themselves
often consist of members originating from different home domains, which, if the expectations are
not managed correctly, can lead to load imbalances and friction. Possible remedies include a) for
identifying the level and needs of the audience, make pre-course surveys, and b) for team projects,
let teams sel-organize into cross-functional teams that fulfill all expectations, identifying
distinct roles and deliverables; let each member take ownership and leadership of their work packages.
Finally, a challenge we identified was transfer of knowledge in teams, transitioning from hierarchical
mentor-mentee structures to structures revolving around teams where members teach and learn from each other.
For this, useful strategies include peer-reviewing of contributions, joint meetings with both
senior and junior team members involved, as well as further mingling activities, as presented, for example,
in retromat.

Well, we need to find a place in the papers somewhere... Maybe it would be a good idea, to put it into the appendix for the institutions paper. I'm still struggling with good ideas, of where to connect it to. It would fit as an introduction to the Master's program of the competencies paper. It would fit into the institutions paper with the intent of making the point, that also our teachers should have a broad background. It would fit into the call-to-action with the intent of describing the current state.... So I guess, putting it to a follow-up issue would be a good idea anyway?

I think current day challenges and lessons could also be a good entry point into institutionalized education
since it's the current state of affairs and not ideal in some to many regards.


Correctly handle global options provided in the option=value form

As identified in #47, global options are not removed from the command options when provided in the form option=value. This results in an invalid option error when parsing the command specific options.
Proposed in this PR is an implementation to fix this.
Fixes #47.

Looks great! Please rebase this branch off latest master and squash the commits to clean up the history and then I think this is ready to merge.

Squashed in c966a85.

@ggilder I think this one is good to go 😄

👍

Released in v4.5.2


vkWaitForFences falsely fails validation when fed a SIGNALED fence

This looks like an oversight - there are many places where fences need to be entered in unsignaled, but this is definitely not one of them.
DBG: MEM: vkWaitForFences specified fence 0xa already in SIGNALED state.
DBG: ParameterValidation: vkWaitForFences: returned VK_ERROR_VALIDATION_FAILED_EXT, indicating that API validation has detected an invalid use of the API

(If I were to "work around" this by calling vkGetFenceStatus before calling vkWaitForFences it would most likely result in a race condition.)

I think this is what is going on looking at the most recent code in master branch.
The first message (DBG: MEM: vkWaitForFences....)  is an INFO level message that is logged.  This indicates something might be wrong with your app but what you are doing is certainly allowable  within the scope of the Vulkan spec. I would suggest only using VK_DEBUG_REPORT_ERROR_BIT_EXT in your VkDebugReportCallbackCreateInfoEXT struct when calling vkCreateDebugReportCallbackEXT() if you wish to only see messages indicating a spec violation.
The second message is triggered by the first message and the debug_report callback that is registered (created) returning true (PFN_vkDebugReportCallbackEXT returns a VkBool32). This
indicates to the validation layer it should abort (not call into the driver) with the current Vulkan command. Typically returning false from your debug_report callback is the thing to do to let the call go to the driver as it normally would even when validation has detected an issue.
The second message

I've since made the debug callback return false if the only bit set is the information bit, but thanks for the note about checking the error bit. Should I close this?

Closing is good by me.


gRPC server sockets going into CLOSE_WAIT state forever on frequent connection/disconnection through gNMI sessions

What version of gRPC and what language are you using?
gRPC 1.51.1, C++
What operating system (Linux, Windows,...) and version?
Linux 5168 5.10.147-oneos-standard https://github.com/grpc/grpc/pull/1 SMP PREEMPT Mon Nov 21 04:26:32 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux
What runtime / compiler are you using (e.g. python version or version of gcc)
g++ (GCC) 4.8.5 20150623 (Red Hat 4.8.5-44.0.3)
What did you do?
Steps:

Open 10 gRPC session by sending a gNMI subscription request from a gNMI client (like go based gnmi_cli, gnmic)
Let the data flow between client and server for 10 sec (This step may not be required)
Terminate all the 10 gRPC sessions in one go like (killall gnmic)
Repeat the step 1 to 3 for few hours.

What did you expect to see?

I expected all the sockets to get cleaned up everytime the connections are terminated. No residuals present and no socket in CLOSE_WAIT state.
New gRPC sessions and in turn gNMI subscriptions should seemlessly work as it used to work in the beginning of this experiment.

What did you see instead?

Initially the cleanup of server sockets was working fine at gRPC server side. I could only see a listening socket at gRPC server side waiting for new requests once all the previous requests were terminated as described in above steps.
After running this in a script for few hours, the gRPC server stopped acknowledging requests and instead threw TCP accept failure log as below:

<155>1 2023-03-03T05:09:03.289565+00:00 telemetry [tcp_server_posix.cc:205] Failed accept4: Too many open files

All the subsequent connections started going into socket CLOSE_WAIT state.
Make sure you include information that can help us debug (full error message, exception listing, stack trace, logs).

Anything else we should know about your project / environment?
Go based gnmi_cli used as client to send subscription request - https://github.com/openconfig/gnmi/blob/master/cmd/gnmi_cli/gnmi_cli.go
Alternatively, you can use gnmic tool from here - https://gnmic.kmrd.dev
Sample gNMI subscription request is as below:
gnmic subscribe -e "proto" -a :9339 -u  -p  --tls-key <key.pem> --tls-cert <cert.pem> --skip-verify sub --path ""

Thank you for the data. I'm not sure the issues are the same, despite the similar symptoms. For example, earlier versions of gRPC had an fd leak with the epollex poller, which was removed in v1.46. It may be too late to diagnose some of those years-old issues.
Without a minimum reproduction case to look at for your instance, it might be hard to determine why there's an fd leak in your application. It could be a usage issue, something preventing endpoints from being properly destroyed, or possibly a new channel is being created every time (inadvisable, partially because of internal caching of established connections). It could also be a gRPC bug.
You may also want to consider a heap profile to see what objects are being leaked, and setting a lower fd limit on the process to trigger the problem sooner.

After lot of testing in different scenarios, found an issue in gnmi application code where sessions are leaking, in a race condition.  After fixing the same, not seeing this issue, so closing this issue now.
Thank you everyone for taking a look.


Completed searchbox funcionality

:tada: This PR is included in version 1.0.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Avatar path not working if hugo page is in subdirectory

This fixes avatar location. Originally it worked only if the page is in the root dir of the domain, now it works if we put it in the subdirectory. It respects baseurl setting in config file.

@b4d could you please squash your commits

I have made some modifications on the theme after that, maybe it would be the best if I close the request and you just fix the avatar path by hand, so I don't interfere with original theme.


Ignore transition and animation from declaration-block-no-redundant-longhand-properties rule

It is especially helpful when adding CSS transitions or keyframe animations to use longhand properties.
We should be able to ignore transition and animation and I think that would be enough: https://stylelint.io/user-guide/rules/declaration-block-no-redundant-longhand-properties/#optional-secondary-options

Thanks for filing this Gerardo! I agree.
To be honest I'd also be fine disabling this rule altogether for a few reasons:

When using custom properties the shorthand can get really long
Sometimes the short hands can get confusing for properties with lots of values (e.g. the font example in the "valid patterns" section of the docs)
I honestly have to spend time thinking every time I see margin: 1px 2px 3px 4px to remember which value affects which side.

That might just be me though 🤷

I'm totally cool disabling it all together. 👍🏽

Disabling it altogether is probably the best option if we're finding ourselves disabling it a lot. Linting is only helpful if it's, you know, helpful. ;)


Change command modifier for MacOS to Cmd

This addresses https://github.com/onivim/oni2/issues/263
I had a hard time writing test cases for this, mainly because I didn't really know how to create a Key-Event object to pass to the corresponding method. Since the rest of Input isn't tested so far, I assume, others had similar issues?
I'd be happy for feedback either way, since this is my first contact with real world Reason.

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.Max Dietrich seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.

Thanks for the PR!
I think leaving the tests for this should be fine, we can add tests for all of the Input in one go.
It looks like there is a CI failure, but you should be able to fix that by running esy format and then pushing up the changes.
I've stuck one comment on the PR as well.

Thank you for the contribution @maxdietrich (and thanks @CrossR for the review)!
Just tried it out on Mac, Win - works great! Glad to have a precedent for this.

I had a hard time writing test cases for this, mainly because I didn't really know how to create a Key-Event object to pass to the corresponding method. Since the rest of Input isn't tested so far, I assume, others had similar issues?

Agree - thank you for thinking about this. IMO we should add an API to Revery to make it easy to generate key events - logged an issue on Revery here: https://github.com/revery-ui/revery/issues/482
Cheers & thank you again - bringing this in now 👍


Update Egnyte.EgnyteDesktopApp.installer.yaml

The SHA256 should be 4CD36AEE6A6F748F2303D9A42D7F11902B036C2F0A11DF6C1F2ED791796B1B49
Checklist for Pull Requests

[x] Have you signed the Contributor License Agreement?
[x] Is there a linked Issue?

Manifests

[x] Have you checked that there aren't other open pull requests for the same manifest update/change?
[x] This PR only modifies one (1) manifest
[x] Have you validated your manifest locally with winget validate --manifest <path>?

[x] Does your manifest conform to the 1.6 schema?

Note: <path> is the name of the directory containing the manifest you're submitting.

Microsoft Reviewers: Open in CodeFlow

ItzLevvie I was checking the 3.17.1 SHA256 , im not sure about 3.17.2


Some answers might be coming from the internet

Please provide us with the following information:

This issue is for a: (mark with an x)
- [X] bug report -> please search issues before submitting
- [ ] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

I asked when Microsoft was founded and was told 1975 with a reference to the 10k Then I asked again and this time was told a specific date April 4, 1975. But this date does not appear in any of our sample data docs.

Any log messages given by the failure

Expected/desired behavior

OS and Version?

Windows 7, 8 or 10. Linux (which distribution). macOS (Yosemite? El Capitan? Sierra?)

Versions

Mention any other details that might be useful

Thanks! We'll be in touch soon.

Updated prompt accordingly.


Add LGTM.com code quality badge to README.md

After #6 is merged, I suspect that the code quality of this repository jumps to A (or even A+)

Of course feel free to reject this PR if you don't like badges!

Coverage remained the same at 100.0% when pulling 4f1bfce4aaa78f695639343fa8eacfae5bacb96f on sjvs:patch-1 into be254899e6bb14ab02d5e9b4a371ae2329c83bef on pim-book:master.


Example 3 fails

After adding debug check (PR https://github.com/mpizenberg/elm-pubgrub/pull/2) Example 3 fails with following output:
> import PubGrub
> import Version
> PubGrub.solve "root" Version.one

Add the following 1 incompatibilities from dependencies of root: ""
   root: 1.0.0, foo: Not ( 1.0.0 <= v )  <<<  from dependency of root at version 1.0.0: ""
Decision level 1 : root : 1.0.0: ""
Derivation : foo : 1.0.0 <= v: ""
Add the following 1 incompatibilities from dependencies of foo: ""
   foo: 2.0.0, bar: Not ( 1.0.0 <= v < 2.0.0 )  <<<  from dependency of foo at version 2.0.0: ""
Decision level 2 : foo : 2.0.0: ""
Derivation : bar : 1.0.0 <= v < 2.0.0: ""
Add the following 1 incompatibilities from dependencies of bar: ""
   bar: 1.0.0, foo: Not ( 1.0.0 <= v < 2.0.0 )  <<<  from dependency of bar at version 1.0.0: ""
Decision level 3 : bar : 1.0.0: ""
Derivation : bar : Not ( 1.0.0 ): ""

Err ("priorCause relation to partialSolution is Satisfies, expected AlmostSatisfies @ PubGrub.continueResolution")

Comparing this to the exact steps algorithm should take:

elm-pubgrub uses incorrect decision level numbers
at Step 4 first term should be foo >=1.0.0 but elm-pubgrub has foo 2.0.0

Thanks for all this @malaire ! I'll join the effort as soon as I can take some time, before weekend if I can, in the weekend otherwise.

I think I've found a bug in PubGrub.makeDecision:
When a version is picked by pickPackageVersion, that picked version is then used as first Term in depIncompats. For example in this Example 3 version 2.0.0 of "foo" is picked, and then depIncompats has foo 2.0.0 as first Term.
BUT that is incorrect because PubGrub algorithm description says in Decision Making that:

Pubgrub collapses identical dependencies from adjacent package versions into individual incompatibilities. This substantially reduces the total number of incompatibilities and makes it much easier for Pubgrub to reason about multiple versions of packages at once. For example, rather than representing foo 1.0.0 depends on bar ^1.0.0 and foo 1.1.0 depends on bar ^1.0.0 as two separate incompatibilities, they're collapsed together into the single incompatibility {foo ^1.0.0, not bar ^1.0.0}.
The version ranges of the dependers (foo in the example above) always have an inclusive lower bound of the first version that has the dependency, and an exclusive upper bound of the first package that doesn't have the dependency. if the last published version of the package has the dependency, the upper bound is omitted (as in foo >=1.0.0); similarly, if the first published version of the package has the dependency, the lower bound is omitted (as in foo <2.0.0). Expanding the version range in this way makes it more closely match the format users tend to use when authoring dependencies, which makes it easier for Pubgrub to reason efficiently about the relationship between dependers and the packages they depend on.

So instead of using that foo 2.0.0 directly, it needs to be converted into a range foo >=2.0.0 which is then used as first Term of depIncompats.

I'll check that this weekend but the first thing coming to mind is that merging incompatibility ranges is only a matter of efficiency and should not change the behavior

One interesting consequence of this is when you can change from single version to any version. For example just before Example 3 fails elm-pubgrub uses bar 1.0.0 as first Term instead of bar any as it should be. I havn't checked this fully yet, but this can be significant.

The any you mention makes me think of that question I asked a while ago to Pub maintainers: https://github.com/dart-lang/pub/issues/2485 where I was wondering how to categorize some incompatibilities. Just had an answer yesterday and didn't take time to process yet but maybe it's related?

Reading about Step 9 of this Example 3, a key point there is that {bar ^1.0.0} satisfies bar any, i.e. when {bar ^1.0.0} is true, bar any must also be true.
Now if you have bar 1.0.0 instead of bar any, that doesn't hold anymore. {bar ^1.0.0} being true doesn't imply that bar is exacly 1.0.0 UNLESS of course 1.0.0 is the only version in that range.
In Example 3 it is the only version, but I don't see PubGrub.conflictResolution accessing Stub.listAvailableVersions so that it can know that in this case {bar ^1.0.0} does satisfy bar 1.0.0 even through generally that would not be so.

Even though the PR that created this issue was closed as incorrect, I think it's worth checking whether bar any case is handled correctly here.
Referring to my comment above, I think algorithm can work correctly without expanding version ranges, but only if it uses the list of available versions to correctly check relations.
That check is done by PartialSolution.relation and Incompatibility.relation, and trying to follow the algorithm I don't see it accessing list of available versions anywhere, so I can't see how it can work correctly with non-expanded versions?

Yep definitely worth checking. Also I think now that I've had a month break from this, a second re-read of the whole thing this weekend will do some good (crossing fingers).


[Feature Request] support enabling and disabling vulnerability alerts

Github (now) supports enabling and disabling Githubs vulnerability alerts settings on the repository.
The upstream golang library has merged support for this as of v25.1.1 and can be pulled in via #228.
It would be really great if we could enable this.

I'll give it a try on this one.

Hi @biancarosa. We would be happy to accept a PR for this if you would like to open one. Looking forward to it, and let us know if you have any questions!

I don't know if this should really a separate issue, but you can also grant more people access to your security alerts. It'd be nice if you can also do that.

@biancarosa are you going to work on this? I also need it for my Github automation.

Please see https://github.com/terraform-providers/terraform-provider-github/pull/382 . I hope it's okay I added the PR right away.
@biancarosa If you have meanwhile worked on a solution, I'm happy to drop my PR and go ahead with yours.

@jtsaito - thanks for reviving this one.
Regarding the proposed implementation, I am in favour of this being a boolean on the existing repository resource, but will defer to what the majority of users feel about the configuration experience.  I think this issue would benefit from another PR that leverages a boolean instead of a new top-level resource.
Lets give @biancarosa until next week to respond as well as some time for the community to chime in on how this implementation should progress.

Submitted a PR with vulnerability alerts as an attribute on the repo resource: https://github.com/terraform-providers/terraform-provider-github/pull/444 .


Slow indexing with Ruby Chewy gem via ActiveRecord

We use ES for analytics but we seem to have some indexing performance issue, and here are some details

We have a MySQL database that stores the processed data
We use a Ruby gem called Chewy (DSL) made by Toptal to interact with ElasticSearch
We index with the Chewy gem to import data from MySQL to ElasticSearch

The issue we are having here is slow indexing performance. Our current indexing is about 5000 MySQL entries per second and it sounds like we are way behind. At this point we are not 100% sure whether it's the MySQL/ActiveRecord/Chewy or ElasticSearch that is the bottleneck. Some diagnostics and profiling we did include:

Running iotop to monitor the disk read/write. It's usually 1MB/s read for a few seconds then a burst of 10MB/s disk write. This makes me think that it could be the read stream from MySQL that's slowing things down.
Separate the read and write stream by connecting Chewy to another MySQL server that is isolated from the ElasticSearch indexing cluster. There is improvement but it was not very significant. It probably went up by 2-3K records/second but still slow.

Since there are many pieces in this indexing process we'd appreciate some help or insights into why the indexing is slow or maybe some pointers about diagnosing the problem.
Some configs about ElasticSearch:
bootstrap.mlockall: true
script.groovy.sandbox.enabled: true
action.disable_delete_all_indices: true.
discovery.zen.ping.multicast.enabled: false

# Heap size defaults to 256m min, 1g max
# Set ES_HEAP_SIZE to 50% of available RAM, but no more than 31g
ES_HEAP_SIZE=8g

# Maximum number of open files
MAX_OPEN_FILES=65535

# Maximum amount of locked memory
MAX_LOCKED_MEMORY=unlimited

# Maximum number of VMA (Virtual Memory Areas) a process can own
MAX_MAP_COUNT=262144

Some MySQL configs:
# This will be passed to all mysql clients
# It has been reported that passwords should be enclosed with ticks/quotes
# escpecially if they contain "#" chars...
# Remember to edit /etc/mysql/debian.cnf when changing the socket location.
[client]
port            = 63306
socket          = /var/run/mysqld/mysqld.sock

# Here is entries for some specific programs
# The following values assume you have at least 32M ram

# This was formally known as [safe_mysqld]. Both versions are currently parsed.
[mysqld_safe]
socket          = /var/run/mysqld/mysqld.sock
nice            = 0

[mysqld]
#
# * Basic Settings
#
user            = mysql
pid-file        = /var/run/mysqld/mysqld.pid
socket          = /var/run/mysqld/mysqld.sock
port            = 63306
basedir         = /usr
datadir         = /var/lib/mysql
tmpdir          = /tmp
lc-messages-dir = /usr/share/mysql
skip-external-locking

# Instead of skip-networking the default is now to listen only on
# localhost which is more compatible and is not less secure.
bind-address            = <IP_ADDRESS>
#
# * Fine Tuning
#
key_buffer              = 128M
max_allowed_packet      = 128M
thread_stack            = 1536K
thread_cache_size       = 64

# * Query Cache Configuration
#
query_cache_limit       = 1M
query_cache_size        = 16M

[mysqldump]
quick
quote-names
max_allowed_packet      = 256M

[mysql]
#no-auto-rehash # faster start of mysql but no tab completition

[isamchk]
key_buffer              = 16M

Environment information:

3 nodes in the ES cluster with 16GB RAM 384GB SSD and Quad Core Intel Xeon CPU
Ubuntu Linux 14.04
ElasticSearch 1.4.4 (We have some dependency issues so haven't upgraded to 2.0+)
Ruby 2.2.2, Rails 4.2, latest Chewy gem

Chewy model example:
define_type ::TypeName.includes(:type_table_from_mysql) do
  field :book_id
  field :entry_time, type: 'date', value: -> { timestamp * 1000 }
  field :in_stock, type: 'boolean', value: -> (o) { ChewyHelper.in_stock?(o) }
  field :price, type: 'float'
  field :day, type: 'integer', value: -> { Time.at(timestamp).utc.yday }
  field :hour, type: 'integer', value: -> { Time.at(timestamp).utc.hour }
  field :date, type: 'integer', value: -> { timestamp / (24*60*60) }
end

Well, unfortunately I've got no idea what's going on, but on your place I would try to write a simple script witch fetches mysql records in batches, manually forms bulk index request body and pushes it to ES. So you will be able to compare chewy performance with pure ruby stuff. And then continue investigation depending on the results

For anyone interested, I first tried with Chewy's "crutches". Barely any speed increase although it was making batch queries, which makes me think my DB wasn't the bottleneck.
Then tried to just "build" the data myself and pass it to Chewy.client.bulk method. This was ~12x faster, than using chewy's import. I guess you lose nice stuff like deleting indexes that no longer exist etc, that (I think) Chewy does for you. But doing it manually at least gave me a chance to build it from the ground up real fast and I can now use Chewy to do updates.
Would be nice if import had a better way to parallellize the work. I kept looking at the sidekiq strategy, but that only seems to work for a block of code. Not sure, but it seemed like it would wait until all indexes are built before posting to ES when importing which doesn't seem like a good idea.

Interesting, thanks, I'll try to find a time to take a look on import performance. Also take a look on this PR https://github.com/toptal/chewy/pull/321 maybe it will be helpful

Hey, could you try to use https://github.com/toptal/chewy/issues/346#issuecomment-198859183 solution? And tell me was it helpful or not.

@kozze89
Hi, there is the same problem in my import application, I intend to try to write a script to manually build the data, but I am kind of busy now. Could you send me your code of buiding data thanks a lot

I am gonna try @dang-hoang-hieu 's thread and a few other options mentioned above and report back here. Thanks for the updates so far.

Hi @pyromaniac and @dang-hoang-hieu  thanks for your inputs on this thread. I finally got some time to actually try different strategies out and here are some reports:
I created a profiling environment to test different strategies:

Always import 30000 records on the BooksIndex::Book
Every time after an import will delete the entire index.

The machine settings are the same as posted in the very first thread on top.
First run without WitchCraft and 0.82 version of chewy, no other options:
WifiIndex::Occupancy Import (28896.6ms) {:index=>30000}
This pass took 28.984829957 seconds

tried witchcraft and 0.8.4 this is like installing a twin turbo to your engine:
WifiIndex::Occupancy Import (16169.6ms) {:index=>30000}
This pass took 16.313680691 seconds

with replication set to async:
WifiIndex::Occupancy Import (16398.2ms) {:index=>30000}
This pass took 16.50799948 seconds

with bulk_size set to 10 MB:
WifiIndex::Occupancy Import (18619.4ms) {:index=>30000}
This pass took 18.719451124 seconds

with bulk_size set to 100 MB:
WifiIndex::Occupancy Import (17861.9ms) {:index=>30000}
This pass took 17.936323542 seconds

turn bulk_size back off to confirm:
WifiIndex::Occupancy Import (16221.1ms) {:index=>30000}
This pass took 16.250417875 seconds

with refresh index after import disabled (however I am not sure if we'd still need to do an index refresh; sounds like we need to, but if this could speed up the indexing and flushing out the index all at once doesn't take too much it might be helpful. This is not improving too much so will leave it as is for now)
WifiIndex::Occupancy Import (15457.3ms) {:index=>30000}
This pass took 15.547787851 seconds

with batch_size set to 30000 (which is the total record I put in the profile):
WifiIndex::Occupancy Import (15928.1ms) {:index=>30000}
This pass took 16.032695959 seconds

Fun discovery:
a. the batch read from MySQL seemed to take less than a second initially - seems like all records were very quickly loaded into mem (iotop confirmed read at first and it was a burst, followed by 0 reads)
b. in the 15 seconds or so, after the MySQL read, there is almost no disk write until by the end there is a burst write of about 15MB/s
c. a&b makes me think it really is chewy/elasticsearch that is using the CPU and mem to do the indexing. If we'd like to improve anything it should be in this area
Next try sidekiq and hopefully we can scale horizontally like it should be:
So scheduled the job with Sidekiq and got this result:
2016-03-28T02:29:27.568Z 10579 TID-w57ss Chewy::Strategy::Sidekiq::Worker JID-6b9714c96015715bf04e3c09 INFO: start
2016-03-28T02:29:42.974Z 10579 TID-w57ss Chewy::Strategy::Sidekiq::Worker JID-6b9714c96015715bf04e3c09 INFO: done: 15.406 sec

Not impressive. Looking into more options with sidekiq
Tried to schedule 6 jobs to run 1..30000, 30001..60000, etc.
2016-03-28T04:10:17.536Z 8143 TID-1jjmcc Chewy::Strategy::Sidekiq::Worker JID-c69ec2284dafc81c906ec968 INFO: done: 94.38 sec
2016-03-28T04:10:17.571Z 8143 TID-1jjnn0 Chewy::Strategy::Sidekiq::Worker JID-4e01c53e7bfd12fced8fd667 INFO: done: 94.328 sec
2016-03-28T04:10:19.868Z 8143 TID-1jjllo Chewy::Strategy::Sidekiq::Worker JID-f086687a88034fc57d694164 INFO: done: 96.614 sec
2016-03-28T04:10:30.972Z 8143 TID-addfo Chewy::Strategy::Sidekiq::Worker JID-8fe04022bcccf118dd97d2e3 INFO: done: 97.144 sec
2016-03-28T04:10:32.193Z 8143 TID-aeu4c Chewy::Strategy::Sidekiq::Worker JID-e1e1125bac72b42427ef3396 INFO: done: 98.344 sec
2016-03-28T04:10:36.815Z 8143 TID-1jhf5s Chewy::Strategy::Sidekiq::Worker JID-5dd9dc9aa18ea9da594f8810 INFO: done: 75.69 sec

The longest job took 98 seconds so that's 98 / 6 ~= 16 second / job
So that basically means there isn't too much concurrency going on here. :(
QUESTIONS
So I wonder if I used sidekiq right:
Chewy::Strategy::Sidekiq::Worker.perform_async("BooksIndex::Book", [*1..30000])

Second question, what are some of the problems that you guys see can be limiting the indexing performance?
Any input would be greatly appreciated and helpful! Thx!

@yang-rifiniti

You should try other options of batch_size, for me:

100 4  7.129461
500 4  2.86017255
1000 4  2.5480937999999997
1500 4  2.144869116666667
2000 4  2.1578910374999998
4000 4  2.113104736111111
10000 4  3.2175962625000003
100 6  7.472030833333334
500 6  2.8385000000000002
1000 6  2.3627573333333336
1500 6  2.1524820222222223
2000 6  2.026925375
4000 6  1.9366167041666666
10000 6  2.0879659150000003
5000 6  2.532585346666667
6000 6  4.464494141666667
7000 6  1.8818611911111112
8000 6  1.841594475
9000 6  1.7835248933333332
10000 6  2.1051081116666666
11000 6  1.9825957444444444
12000 6  2.1013876277777777

It turned out that 9k with 6 processes took 1.78 hours
2. If you try sidekiq, you can use multi-process like with 4 processes, you create 4 new terminal and then run 4 sidekiq instance with thread = 1 (you can see in sidekiq web 4 processes with 4 threads)
3. Or you can spawn process directly from code
I will update code spawning process later :)

Hey @dang-hoang-hieu thx for the quick response. I figured out the big picture idea of sidekiq but since I am still pretty new to it if you can share w me the code (thx in advance for willing to share the spawning process too!), that'd be very very helpful!

yep

for sidekiq, you create a new ternimal tab, run sidekiq instance with thread = 1. Then create another terminal tab, run sidekiq instance with thread = 1, each tab is a process
code for spawn, rake task:

namespace :reindex do
  task candidate: :environment do
    logger = ActiveSupport::Logger.new("log/reindex.log")
    total_doc = 0
    count = 0
    start = Time.now
    Candidate.find_in_batches(batch_size: ENV["BATCH_SIZE"].to_i) do |candidates|
      pid = fork {
        CandidatesIndex::Candidate.import candidates, suffix: start.to_i
      }
      count += 1
      if count == ENV["POOL_SIZE"].to_i
        Process.waitall
        elapsed = Time.now - start
        total_doc += ENV["POOL_SIZE"].to_i * ENV["BATCH_SIZE"].to_i
        log = "process: #{total_doc} after #{elapsed / 60} minutes"
        logger.info log
        puts log
        count = 0
        break if ENV["TEST"]
      end
    end
    logger.close
  end
end

I run
bundle exec rake reindex:candidate POOL_SIZE=4 BATCH_SIZE=9000

Well, I personally would prefer Chewy::Strategy::Sidekiq::Worker.perform_async("BooksIndex::Book", candidates) instead of forking. But yeah, a kind of :)

@pyromaniac thanks for your hint. I wonder what the difference is between @dang-hoang-hieu 's approach and the perform_async approach? Is the perform_async a wrapper for the same thing as @dang-hoang-hieu wrote right above?
I tried the perform_async but didn't see indexing improvement (about 2K records/s) but w @dang-hoang-hieu 's method I got a solid 4K records/s which is a bless for me. Could it be when I use the perform_async I didn't set up some threading params right or?
ES 2.x Support
@pyromaniac Wonder if the latest chewy gem supports 2.x and if it is not currently supported, is it on the road map? I spoke w ES people in their Chicago office before and they recommended 2.x for stability and added features but not sure if chewy will support. Any thoughts or plans would be helpful.
** data stability **
So in our use case we do a full indexing by doing a BooksIndex.reset! and if everything goes through in one pass the data is usually very solid. However if we delete one period of data (e.g. 2013-2014 data entries), and reindex some other periods of time (2013/05 - 2014/01) then sometimes the queries would fail with random errors (such as parsing error, out of memory, etc.) Wonder what could be causing that if you can offer any pointers. We are still on ES 1.4.4, could that be the problem? Any best practice kind of advice would be greatly appreciated!

Reporting back some updates here:
With a pretty beefy machine I've got:
POOL_SIZE=4 BATCH_SIZE=9000; 5769 records / s
POOL_SIZE=9 BATCH_SIZE=9000; 10242 records / s
POOL_SIZE=13 BATCH_SIZE=9000; 11470 records / s
I think tweaking the numbers can potentially make it even faster as I still occasionally see 50% on all cores CPU time.

The difference is that you are putting all the records in one job: Chewy::Strategy::Sidekiq::Worker.perform_async("BooksIndex::Book", [*1..30000]), but you should do it like Chewy::Strategy::Sidekiq::Worker.perform_async("BooksIndex::Book", [*1..1000]); Chewy::Strategy::Sidekiq::Worker.perform_async("BooksIndex::Book", [*1000..2000]). Only in this case it will work as expected.
Chewy supports 2.0, but still uses deprecated features. I'm going to work on this definitely.
As for data stability would be better to post errors themselves. Also upgrade to 1.7.5.

@pyromaniac right that makes perfect sense. thx for the quick update!

@yang-rifiniti (y)
you can try more processes to get the best performance
my server only has 6 cores, it works best at 10 processes :)

Yep we did a 12 core w 32GB ram and 24 threads; close to 15K records/s never been this happy!


CreateClaimableBalance is not within the allowed operations

Precheck

[x] Do a quick search and make sure the bug has not been reported yet.
[x] Please disclose security vulnerabilities privately at<EMAIL_ADDRESS>[x] Finally, be nice and have fun!

Description
CreateClaimableBalance is not within the allowed operations.
Environment

Elixir & Erlang/OTP versions (elixir --version): Elixir 1.13.4 (compiled with Erlang/OTP 24)
Operating system: Linux Ubuntu 22.04 LTS

Current behavior
When you create a transaction that includes the CreateClaimableBalance operation, you get a response that indicates that the operation is unknown.
{:error, [unknown_operation: %Stellar.TxBuild.CreateClaimableBalance{...}]}

Expected behavior
It should return {:ok, tx} as the library response because CreateClaimableBalance is a valid operation in the Stellar network.

Fixed in the version 0.10.1 👌🏻


Segmentation Fault error when I add sqlsrv driver

I am trying to implement an RFC call to fetch some data and then, query to a SQL Server to do some filtering.
I can do the RFC Call and fetch the data without any problem when the pdo_sqlsrv.ini and sqlsrv.ini files are not in /etc/php/8.0/mods-available directory.
When I add those two to the directory I get  this error:
PHP Warning:  Module "pdo_sqlsrv" is already loaded in Unknown on line 0

Violación de segmento

The sqlsrv drivers work and I can get data from the databases but it seems like there is a incompatibiliy between the sapnwrfc and pdo_sqlsrv or sqlsrv drivers. I am guessing It could be something about data types...
The sql drivers installed are the 5.9.0 version. PHP client version is the 8.0.8.

I re-installed the latest version in the main repository and now i get this error instead of the Segmentation Fault...
PHP Warning:  Module "pdo_sqlsrv" is already loaded in Unknown on line 0

Array
(
    [code] => 22
    [key] => RFC_CONVERSION_FAILURE
    [message] => Illegal input during conversion from RFCTYPE_FLOAT to RFCTYPE_BCD
)

If I remove the sqlsrv files from the /etc/php/8.0/mods-available directory I can still fetch the data without any problem....
I add them again and I get this error.

Please provide a backtrace when you encounter a Segmentation fault.

The error happens when I call the function invoke to get the custom function module (If I call the STRF_CHANGING module it works).
I am sorry If that's not the answer but I don't really know what you mean by "backtrace". Do u want me to show a specific log or try a specific code?

PHP Warning:  Module "pdo_sqlsrv" is already loaded in Unknown on line 0

This means you are trying to load the module twice. Check your PHP configuration files.
As for the conversion error, I will need additional information. Please provide the exact version of the sapnwrfc extension (git revision if not a release) and a complete, self-contained example for me to try and reproduce (PHP code and a small ABAP function module if necessary).
If you encounter a Segmentation fault, you will have to use a debug build of PHP and the extension and then extract a backtrace from the coredump using gdb. Google is your friend on how to do that.
Note that PHP 8 support is still experimental at the moment.

I am using the last version of the sapnwrfc extension with the php 8.0 support.
The php code i'm trying to run:
	use SAPNWRFC\Connection as SapConnection;
	use SAPNWRFC\Exception as SapException;

	// Función que devuelve todos los pedidos realizados de una planta en un rango de fecha [dtFechaInicio - dtFechaInicio] (fmt: YYYYMMDD)
	function rfcPedidos($strPlanta, $dtFechaInicio, $dtFechaFin)
	{
		// definimos los parametros de conexion
		// TODO: Los parametros vienen de la BDD.
		$config = [
			'ashost' => 'XXXX',
			'sysnr'  => 'XX',
			'client' => 'xxx',
			'user' => 'xxxx',
			'passwd' => 'xxxxx',
			'lang' => 'ES',
			'trace'  => SapConnection::TRACE_LEVEL_OFF,
		];

		$parameters = [
			"P_WERKS" => $strPlanta,
			"P_BUDAT_LOW" => $dtFechaInicio,
			"P_BUDAT_HIGH" => $dtFechaFin
		];

		//$f = $c->getFunction('ZPE_MM_EXT_POS_PEDIDOS_MECAL');

		try {

			// creamos la conexion RFC

			$c = new SapConnection($config);
			$f = $c->getFunction('ZPE_MM_EXT_ALBAR_MECAL');

			$result = $f->invoke($parameters);
                        // fails here...

			//var_dump($result);

			// array que contendra todos los repartos correspondientes a sus posiciones y cabeceras.
			$pedidos = [];

			// Mostrar datos de las diferentes tablas que devuelve el SA
			foreach($result["I_HISTORIAL"] as $pedido){

				$a = array(
					"Numero de pedido" => $pedido["EBELN"],
					"Pos pedido: " => $pedido["EBELP"],
					"Año: " => $pedido["GJAHR"],
					"Documento SAP: " => $pedido["BELNR"],
					"Fecha contabilización: " => $pedido["BUDAT"],
					"Cantidad: " => $pedido["MENGE"],
					"Coste: " => $pedido["DMBTR"],
					"Albarán: " => $pedido["XBLNR"],
					"Material: " =>  $pedido["MATNR"],
					"Usuario SAP: " => $pedido["ERNAM"]
				);

				array_push($pedidos, $a);

			}

			print_r($pedidos);

			$c->close();
		 } catch(SapException $ex) {
			print_r($ex->getErrorInfo()) . PHP_EOL;
		}

I am no getting a segmentation fault core dump anymore so I don't know how to get the backtrace for that run. I tried this:
(gdb) run server.php
Starting program: /usr/bin/php server.php
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Array
(
    [code] => 22
    [key] => RFC_CONVERSION_FAILURE
    [message] => Illegal input during conversion from RFCTYPE_FLOAT to RFCTYPE_BCD
)
[Inferior 1 (process 11648) exited normally]
(gdb) bt
No stack.
(gdb)

I am also attaching the result of executing phpinfo() command in case it helps.
test.txt

How does ZPE_MM_EXT_ALBAR_MECAL look like? What data types are returned? Have you tried setting lang to EN?

The datatype conversion is the fallowing

It seems that there is an issue with the fallowing datatype...

I tried to change lang to EN and still getting the same error.
What i don't really understand is that when i remove the sqlsrv drivers the conversion works and and I get correctly all the values. The sqldriver affects  somehow the conversion behaviour.

OK, thanks for the information, I'll try to have a look tomorrow evening...

I am unable to reproduce this issue...


stanc: 2.32.2 -> 2.33.1

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://github.com/stan-dev/stanc3/releases.
meta.description for stanc is: The Stan compiler and utilities
meta.homepage for stanc is: https://github.com/stan-dev/stanc3
Updates performed

Version update

To inspect upstream changes

Release on GitHub

Compare changes on GitHub

Impact

Checks done (click to expand)

built on NixOS
The tests defined in passthru.tests, if any, passed
found 2.33.0 with grep in /nix/store/sca3iipp9p8ili1h47i2qmrww7w2icld-ocaml4.14.1-stanc-2.33.0
found 2.33.0 in filename of file in /nix/store/sca3iipp9p8ili1h47i2qmrww7w2icld-ocaml4.14.1-stanc-2.33.0

Rebuild report (if merged into master) (click to expand)

6 total rebuild path(s)

6 package rebuild(s)

First fifty rebuilds by attrpath
cmdstan
python310Packages.cmdstanpy
python310Packages.prophet
python311Packages.cmdstanpy
python311Packages.prophet
stanc

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/sca3iipp9p8ili1h47i2qmrww7w2icld-ocaml4.14.1-stanc-2.33.0 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A stanc https://github.com/r-ryantm/nixpkgs/archive/cc05710463e2fdf10f15d68564fadb47453216f1.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/sca3iipp9p8ili1h47i2qmrww7w2icld-ocaml4.14.1-stanc-2.33.0
ls -la /nix/store/sca3iipp9p8ili1h47i2qmrww7w2icld-ocaml4.14.1-stanc-2.33.0/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

4 packages failed to build:

python310Packages.cmdstanpy
python310Packages.prophet
python311Packages.cmdstanpy
python311Packages.prophet

2 packages built:

cmdstan
stanc

Maintainer pings
cc @wegank for testing.

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://github.com/stan-dev/stanc3/releases.
meta.description for stanc is: The Stan compiler and utilities
meta.homepage for stanc is: https://github.com/stan-dev/stanc3
Updates performed

Version update

To inspect upstream changes

Release on GitHub

Compare changes on GitHub

Impact

Checks done (click to expand)

built on NixOS
The tests defined in passthru.tests, if any, passed
found 2.33.1 in filename of file in /nix/store/qhi8dzxkw310r4av7kcc15hyv2df7b4y-ocaml4.14.1-stanc-2.33.1

Rebuild report (if merged into master) (click to expand)

6 total rebuild path(s)

6 package rebuild(s)

First fifty rebuilds by attrpath
cmdstan
python310Packages.cmdstanpy
python310Packages.prophet
python311Packages.cmdstanpy
python311Packages.prophet
stanc

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/qhi8dzxkw310r4av7kcc15hyv2df7b4y-ocaml4.14.1-stanc-2.33.1 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A stanc https://github.com/r-ryantm/nixpkgs/archive/f84af00d0ed277225c4b62cd633c0c93c70b8370.tar.gz

Or:
nix build github:r-ryantm/nixpkgs/f84af00d0ed277225c4b62cd633c0c93c70b8370#stanc

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/qhi8dzxkw310r4av7kcc15hyv2df7b4y-ocaml4.14.1-stanc-2.33.1
ls -la /nix/store/qhi8dzxkw310r4av7kcc15hyv2df7b4y-ocaml4.14.1-stanc-2.33.1/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

4 packages failed to build:

python310Packages.cmdstanpy
python310Packages.prophet
python311Packages.cmdstanpy
python311Packages.prophet

2 packages built:

cmdstan
stanc

Maintainer pings
cc @wegank for testing.


Sticky Scroll (Ubuntu with Python)

Does this issue occur when all extensions are disabled?: No, but with one specific extension

VS Code Version: 1.75.1 -- 441438abd1ac652551dbe4d408dfcec8a499b8bf -- x64
OS Version: Ubuntu 20.04

Steps to Reproduce:

Enable Sticky Scroll. Loops and conditionals also stick to the top along with class and functions.
Install Python IntelliSense (Microsoft). Only class and functions stick to the top. Loops and conditionals don't.
Uninstall or disable Python IntelliSense and it goes back to normal.

Maybe #174958 will help.

Hi, thanks for your message. As @gjsjohnmurray was suggesting, once the PR is in, you will be able to define the folding model as the default model to use for sticking the lines. The folding model is based on the indentation and will therefore stick also the for loops.

Closing since the PR has been merged


Telecon, December 17th

Link for the call is here: https://washington.zoom.us/j/144327307
Given this might be our last telecon before the sprint, we should  probably use this call to make a rough schedule for the sprint.

12.30 PST, I assume?
cc/ @nipreps/dmriprep @dPys @richford @garikoitz @mattcieslak @josephmje @jelleveraart

Yes. Sorry about the vagueness. I'll be there.

Works for me!

Working on a draft for the agenda here - https://hackmd.io/@SdQZX7wbSj6wewa5sQntwg/Bk46asU0B/edit


fix(express): Use rawBodyMiddleware only when the content-type is app…

…ropriate and the size is under 100kb
Closes: #1598

/rebase

:tada: This PR is included in version 6.75.11 :tada:
The release is available on:

v6.75.11
GitHub release

Your semantic-release bot :package::rocket:


python311Packages.zope-contenttype: 4.6 -> 5.1; rename from zope_contenttype

Description of changes
Changelog: https://github.com/zopefoundation/zope.contenttype/blob/5.1/CHANGES.rst
part of #245383

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[x] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
23.11 Release Notes (or backporting 23.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 260333 run on x86_64-linux 1

4 packages built:

python310Packages.zope-contenttype
python310Packages.zope-contenttype.dist
python311Packages.zope-contenttype
python311Packages.zope-contenttype.dist


New package: ComputedGeodesicEquations v0.1.0

Registering package: ComputedGeodesicEquations
Repository: https://github.com/astro-group-bristol/ComputedGeodesicEquations.jl
Created by: @fjebaker
Version: v0.1.0
Commit: ac8749aad3088adcdf328c7831946defa81534aa
Git reference: HEAD
Description: Generated Julia functions for geodesic equations in common spacetimes.

This can be closed (#48533).
This package was as much to see how package registration works in practice, as it was to start building a dependency network for a project in development. The option of setting up a private repository had not occured to us for such utility dependencies, but is an avenue we will explore.
Thanks!


Fast client writes will fill up memory

The client code using libws has no way to tell how fast data is being written to the socket, so it can't tell how fast it can send messages. If it sends messages faster than the socket can write them, the libevent write buffers will start to pile up and increase memory usage. This also increases latency, since the client can end up creating messages that only get written to the socket much later.
(Think of a webcam that wants to send camera frames over the socket as JPEG images. If it starts sending messages at 60FPS, but the network bandwidth isn't enough to support that, then memory will start to fill up with pending frames, and the other side of the connection will see the video feed in slower than real-time. Instead, the webcam should wait until the websocket is ready for more data, and then write the current frame.)
This can be fixed by adding a new callback that tells the client that the socket has room to write. This is equivalent to the write callback provided by libevent; in fact libws's write callback can invoke the new callback.

By the way, I'm filing these issues partly as a way to probe whether this project might come back to life — it looks promising (compared to the other libevent-based WebSocket implementations I've looked at) but it's seen no activity for 2 years.
I have fixes for all the issues I've filed, currently in a private fork, but I can submit them as PRs if there's interest.

Thank you for pointing out the issues.
This project was a hobby project of mine to learn libevent and the websockets protocol and API design in a library.
I tried to design it in a way so that it is possible to replace Libevent with any other event library/OS api without having to change the external API. Which in some ways makes things more complicated than they need to be if all you are interested in is to use Libevent. Again this was just a challenge for myself.
However as you have noticed there are a lot of issues still remaining. But I have kind of lost interest in the project myself. A big reason being that I don't really use the websocket protocol in any project. I have never used this library for real, other than my test clients (which explains all the issues you have found). When trying to get full SSL support working I kind of gave up. Because of how bad the OpenSSL init API is, I just did not manage to do it in a nice way which made me lose the "fun" part of it all. But apparently the new verisons have a new API that doesn't make it crash because you init it more than once in an application :)
Here is a fork that has done some substantial changes that removes the "hiding" of Libevent and exposes things:
https://github.com/alxvasilev/libws
It seems to still be actively being developed. But I have not gotten any Pull Requests, or had any contact with this dev, so it can be considered a standalone fork.
However, if you have fixes for these issues I am glad to accept Pull Requests and merge them. If you are interested in becoming an active maintainer of the project I'm up for that as well. But I don't want to give you any false hopes that I might start maintaining it actively myself, since I have too many other projects going

Hello from the author of the mentioned fork :)
First of all, a big thanks to Joakim for his neat library. I have reviewed quite a few websocket client libs, and this was was the best for our needs.
We have brought the codebase to a state where it fits our needs and is flexible enought to replace libevent with another library of that type. But at the moment we unfortunately don't have much time to review and accept patches. This may happen in the future, as we actually replace libevent and maybe do another round of fixes/changes to the code. Until then, I am unable to do maintenance of our fork of the library.

@alxvasilev hello, thanks for the reply. I have looked at some of your changes and I like many of your fixes. It's fun to see my hobby project actually has been useful for someone.
But yea there are some parts of the fixes I would not merge right away. So you don't need to do a PR. But maybe I get some motivation to port parts of your changes back upstream some day :) Hopefully in the process making things acceptable for you to use "as is". Specifically I saw you removed the autobahn test suite. I guess since it requires additional dependencies? I find it good to have such tests to verify that any changes still adheres to the Websocket standard.


Aggregated logs

Hi,
View Aggregated Logs allows combined view of only stdout. That's pretty much useless if singularity executor is in use. Is there a way to follow tail of service.log inside of each task instance?

In the ui portion of SingularityService yaml, you can set runningTaskLogPath: $TASK_ID/service.log to update all of the defaults for these types of links to be the service.log instead

Right now this is a global


Upgrade actions/upload-pages-artifact v2.0.0 -> v3.0.0

This PR contains the following updates:

Package
Type
Update
Change

actions/upload-pages-artifact
action
major
v2.0.0 -> v3.0.0

Release Notes

actions/upload-pages-artifact (actions/upload-pages-artifact)
v3.0.0
Compare Source
Changelog

Use v4 upload-artifact tag @robherley (#80)
Upload pages artifact with upload-artifact v4-beta @konradpabjan (#78)

To deploy a GitHub Pages site which has been uploaded with his version of actions/upload-pages-artifact, you must also use actions/deploy-pages@v4 or newer.
See details of all code changes since previous release.

[ ] If you want to rebase/retry this PR, check this box

The Java Platform team wishes you a happy festive period. Here is the suggested commit message:
Upgrade actions/upload-pages-artifact v2.0.0 -> v3.0.0

Renovate be like

Header generated and modified from Sweaterify.


it is not clear how to work with useInfiniteQuery in v2

it is not clear how to work with useInfiniteQuery in v2
ex:
  const {
    error,
    data,
    isFetching,
    isFetchingMore,
    fetchMore,
    canFetchMore,
  } = useInfiniteQuery([roomId, { messageId }], fetchMessagesByRoomId, {
    getFetchMore: lastGroup => lastGroup?.length > 14,
    onSuccess,
  });

how convert this to v2 style???

This should work exactly the same in v2.


Getting an error while installing you package using Atom.

I tried intsalling it varios times but it results in the same error.

In case the photo isn't visible, the error goes like :
fs.js:922
  return binding.mkdir(pathModule._makeLong(path),
                 ^

Error: EPERM: operation not permitted, mkdir 'C:\PROGRA~1\Java\JDK18~1.0_1\bin\d-118624-8948-f5tgww.gqmyhqia4i'
    at Error (native)
    at Object.fs.mkdirSync (fs.js:922:18)
    at Object.mkdirSync (C:\Users\DELL\AppData\Local\atom\app-1.28.2\resources\app\apm\node_modules\temp\lib\temp.js:137:6)
    at Request.<anonymous> (C:\Users\DELL\AppData\Local\atom\app-1.28.2\resources\app\apm\lib\install.js:335:41)
    at emitOne (events.js:96:13)
    at Request.emit (events.js:188:7)
    at Request.onRequestResponse (C:\Users\DELL\AppData\Local\atom\app-1.28.2\resources\app\apm\node_modules\request\request.js:1062:10)
    at emitOne (events.js:96:13)
    at ClientRequest.emit (events.js:188:7)
    at HTTPParser.parserOnIncomingClient [as onIncoming] (_http_client.js:474:21)

Thanks for your help.

Hello you there?

You can out put a list of other installed packages (that way we can see if one is conflicting with this one for example) by using apm (atom package manager cli tool).
$ apm list --installed
and you can just copy and paste the contents there. This may be an issue with atom or with the package. My best guess is it may be Atom and/or a conflicting package seeing as the error seems to be occurring within a package dependency fs.js on line 922.
Sometimes you have to rebuild the module cache which apm can do for you.
$ apm rebuild-module-cache
Sometimes certain JS pacakges need to be recompiled and reconfigured for one reason or another and this can fix that issue. Other than that, I really don't know without more info.
You can check out these links and hopefully they lead to something useful.

Wiki - readme
Logs - console log
Logs - cp log
How to use atom-python-run

You may also want to check out these alternative resources and check out atom.io forums and see if others have faced a similar issue. With some web searching and patience, you find that it might be something else entirely thats keeping you from installing the package.


ignore list

I can't get "ignore" to work on a uri and its sub queries.
"ignore" => ["/list"]
ex:
/list
/list/1
/list/3
the codes are dynamic so it's impossible to put them all

How are you configuring the middleware? Also show an example request and response with curl. For example:
$ curl --include http://www.example.com/list/2
HTTP/1.1 404 Not Found
Cache-Control: max-age=604800
...


linkage_checker: allow linkage via alias

[x] Have you followed the guidelines in our Contributing document?
[x] Have you checked to ensure there aren't other open Pull Requests for the same change?
[ ] Have you added an explanation of what your changes do and why you'd like us to include them?
[ ] Have you written new tests for your changes? Here's an example.

[x] Have you successfully run brew typecheck with your changes locally?
[ ] Have you successfully run brew tests with your changes locally?

Looks good so far but would love any more context as to why?

Specific scenario was from reversing alias on icu4c so uses_from_macos "icu4c" ends up showing indirect linkage warnings.
Still deciding if this is worth it or just ignoring the macOS dependency information and replacing with on_linux (gnustep-base) or on_system :linux, macos: :monterey_or_older (r).
May consider latter in https://github.com/Homebrew/homebrew-core/pull/196166


Evaluate switching graphql server module

The current server seems to not be very actively developed, with the last commit being over 5 months ago.
Additionally, when searching for posts/opinions about graphql servers in go, nobody seems to recommend that one, and usually one of two other servers are recommended:
graphql-go
gqlgen
Both options seems pretty good, so we might want to evaluate switching to one of them at some point

Both of these servers look like they rely on strong (gqlgen) or somewhat-strong (graph-gophers) typing for resolvers.
Since our schema is dynamic and not known at build time, I'm not sure how we can use either of these solutions - gqlgen generates code at compile time, and graph-gophers requires manually writing typesafe resolvers with no plans to change that
OTOH, our current server is very flexible which is great for a project like agogos. Unless we have performance/stability/bug issues I don't see the real appeal in switching to something else.

Sort-of irrelevant now that it's on apollo


Remove Node.js for caskdata/cdap#5522

We no longer need to provide the Node.js packaging as it is included in CDAP RPM and DEB packaging in CDAP 3.4+

LGTM


deps: Fix protoc release date

Signed-off-by: Ryan Northey<EMAIL_ADDRESS>Commit Message:
Additional Description:
Risk Level:
Testing:
Docs Changes:
Release Notes:
Platform Specific Features:
[Optional Runtime guard:]
[Optional Fixes #Issue]
[Optional Fixes commit #PR or SHA]
[Optional Deprecated:]
[Optional API Considerations:]

/retest

/lgtm deps

closing this on the basis that there is now another release date issue (proxy-wasm-host) so i have combined the fix for the checker, and the date fixes into #22504


Add CPP code for connecting N ropes with minimum Cost

@pranjal021 @NyorJa Please Review my PR and consider merging it.


Update cluster client flow

#1098  👈
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @carolineechen and the rest of your teammates on  Graphite


The command npx @thebcms/cli create next starter always downloads the simple-blog template

Hello,
When I try to init the agency starter project with the provided command
npx @thebcms/cli create next starter agency
It downloads the simple-blog instead.
Am I doing something wrong ?

Hey @Brinxx06 ,
Thank you for trying BCMS and opening this issue. You are doing everything correctly. The reason the simple-blog was always created is because we did not add the newly added starters to info.json, and because of that, the CLI did not know that they were ready for use.
The commit below should solve the issue that you were encountering. Just keep in mind that not all starters are yet ready for all frameworks. We are working hard to make them available as soon as possible.
https://github.com/bcms/starters/commit/d0f6307a951208ef2f95986cabbc6428f758452e


HKG: Add FW Versions for 2019 Kia Stinger

Add firmware for 2019 Kia Stinger.
Dongle ID: aa73ed79a1b82fb3
Thanks to community 2019 Kia Stinger owner Meck#5341 (Discord).

Also, I'm not sure I trust the engine FW here, you pulled the FW versions from an old fork of OP where we didn't do adequate fingerprint logging. Can the user run master?